microsoft
/
MLOS
зеркало из https://github.com/microsoft/MLOS.git
1
0
Форкнуть 0
Код Задачи Пакеты Проекты Релизы Вики Активность
MLOS/Makefile

Ignoring revisions in .git-blame-ignore-revs. Click here to bypass and see the normal blame view.

602 строки
29 KiB
Makefile
Исходник Обычный вид История

Merged PR 1054: Prepwork for OSSing - include some boilerplate from the original MLOS repo - adding license headers on scripts - add rules for checking licenseheaders
2023-03-18 00:20:39 +03:00
# Copyright (c) Microsoft Corporation.
# Licensed under the MIT License.
Rename the conda env (#319) * rename the conda env, which serves mlos_bench as much as mlos_core now, to just "mlos" * add rename to the docker container - pulled from #313
2023-04-28 22:13:41 +03:00
CONDA_ENV_NAME ?= mlos
PYTHON_VERSION := $(shell echo "${CONDA_ENV_NAME}" | sed -r -e 's/^mlos[-]?//')
Merged PR 610: Adding more codestyle checkers and fixups pylint apparently removed the bad-whitespace check at some point [1] This adds pycodestyle back to the mix to check for it. [1] https://github.com/PyCQA/pylint/issues/3512
2022-09-02 21:58:49 +03:00
ENV_YML := conda-envs/${CONDA_ENV_NAME}.yml
Merged PR 600: Fixup wheel generation
2022-09-01 00:49:31 +03:00
# Find the non-build python files we should consider as rule dependencies.
Small tweaks (#901) # Pull Request ## Title Small tweaks --- ## Description - mypy fixup - Minor perf improvement on Makefile file enumeration. - Add typevar hints for `nullable` return type. - Fix a minor build system issue - List python 3.13 as one of the supported versions (#879) --- ## Type of Change - 🛠️ Bug fix - 🔄 Refactor --- ## Testing Usual pytest and mypy CI checks for `nullable` --- ## Additional Notes (optional) Split out from #720 ---
2025-01-04 00:55:27 +03:00
# Do a single find and multiple filters for better performance.
REPO_FILES := $(shell find . -type f 2>/dev/null | egrep -v -e '^./(mlos_(core|bench|viz)/)?build/' -e '^./doc/source/' -e '^./doc/build/')
PYTHON_FILES := $(filter %.py, $(REPO_FILES))
MLOS_CORE_PYTHON_FILES := $(filter ./mlos_core/%, $(PYTHON_FILES))
MLOS_BENCH_PYTHON_FILES := $(filter ./mlos_bench/%, $(PYTHON_FILES))
MLOS_VIZ_PYTHON_FILES := $(filter ./mlos_viz/%, $(PYTHON_FILES))
NOTEBOOK_FILES := $(filter %.ipynb, $(REPO_FILES))
SCRIPT_FILES := $(filter %.sh %.ps1 %.cmd, $(REPO_FILES))
SQL_FILES := $(filter %.sql, $(REPO_FILES))
MD_FILES := $(filter-out ./doc/%, $(filter %.md, $(REPO_FILES)))
Merged PR 333: Initial MlosCore Implementation Adds basic `register` and `suggest` APIs for a couple of optimizer backends: `Skopt`, `Emukit`, `Random`. Includes example notebook for starters. Limited unit testing. Basic function documentation. Azure DevOps CI pipelines for `pylint` and `pytest` via the `mlos_core` `conda` `environment.yml`. Related work items: #274, #279
2022-03-14 22:37:52 +03:00
Merged PR 903: Combination of additional build and doc fixups This PR was originally aimed at just fixing the intermittent documentation build errors. That issue ultimately stems from conflicts between conda and pip installs of two different versions of `mistune` dependency. Along the way I found a handful of other improvements for build, pipeline, and documentation generation things. - Make sure we do the nightly build regardless of whether code has changed - Don't build the docs in parallel by default (trying to avoid an intermittent build failure) - Add basic support for testing devcontainer manually outside for vscode - Stop doing parallel builds so it's easier to debug, and move doc build/publishing to the devcontainer - Split windows conda env off, stop using gpy from conda-forge, and just build it locally with clang, gcc, or msvc - Adjust the devcontainer build so that more dependencies are built upfront during the container build. This makes the process more cacheable so that we can just pull the upstream version and not have to build as much locally which could delay first launch. - Import part of a fix from elsewhere - Switch to docker-compose style devcontainer that automatically also starts an nginx container for viewing the generated docs - Update some docs and doc building - [X] pydocstyle fixups - [X] Add some log filtering of the `make doc` output for easier debugging (e.g. don't care about missing examples for now) - [X] doc link lint checks - DONE - [X] Split this out into multiple PRs - [X] SpaceAdapterFactory: !959 - [X] docstyle fixups: !960 - [ ] doc build fixups - [ ] devcontainer improvements (leaving the last two as this PR)
2023-02-27 08:42:22 +03:00
DOCKER := $(shell which docker)
# Make sure the build directory exists.
mypy static type checking for mlos_bench (#306) Adds static type checking via mypy for mlos_bench as well. Builds on #305, #307 - [x] Add `Protocol`s for different `Service` types so `Environment`s can check that the appropriate `Service` mix-ins have been setup. - [x] Config Loader - [x] Local Exec - [x] Remote Exec - [x] Remote Fileshare - [x] VM Ops - [ ] Future PR: Split VM Ops to VM/Host and OS Ops (for local SSH support)
2023-04-19 21:28:38 +03:00
MKDIR_BUILD := $(shell test -d build || mkdir build)
Merged PR 860: Various Build Fixups - add nightly builds so we can catch issues with upstream changes sooner - allow reading conda envs out of the environment - handle only setting one of CONDA_EXPERIMENTAL_SOLVER and CONDA_SOLVER envs - adjust conda envs to avoid python 3.11 for now - adjust Makefile to use the same version of python from the current CONDA_ENV for our test environment (this is quick way to respect that version constraint and only specify it in one place) - adjust the expected version of configspace - remove configspace from conda env file (upstream doesn't have the version we need outside of conda-forge) - move all other version restrictions to setup.py files - add a 'full' mode for installing all extra_requires for each package - use that in the conda envs instead Related to additional build/test issues discovered in !850
2023-01-31 02:49:26 +03:00
Merged PR 572: Document experimental conda solver and enable Windows CI jobs This is a follow up to !557 (Make conda work on Windows) and addresses some of the lingering comments from that PR. - document the mamba slow solver workaround - use the mamba solver if available in Makefile - add windows jobs to the CI tasks as well
2022-08-18 07:43:22 +03:00
# Allow overriding the default verbosity of conda for CI jobs.
Merged PR 903: Combination of additional build and doc fixups This PR was originally aimed at just fixing the intermittent documentation build errors. That issue ultimately stems from conflicts between conda and pip installs of two different versions of `mistune` dependency. Along the way I found a handful of other improvements for build, pipeline, and documentation generation things. - Make sure we do the nightly build regardless of whether code has changed - Don't build the docs in parallel by default (trying to avoid an intermittent build failure) - Add basic support for testing devcontainer manually outside for vscode - Stop doing parallel builds so it's easier to debug, and move doc build/publishing to the devcontainer - Split windows conda env off, stop using gpy from conda-forge, and just build it locally with clang, gcc, or msvc - Adjust the devcontainer build so that more dependencies are built upfront during the container build. This makes the process more cacheable so that we can just pull the upstream version and not have to build as much locally which could delay first launch. - Import part of a fix from elsewhere - Switch to docker-compose style devcontainer that automatically also starts an nginx container for viewing the generated docs - Update some docs and doc building - [X] pydocstyle fixups - [X] Add some log filtering of the `make doc` output for easier debugging (e.g. don't care about missing examples for now) - [X] doc link lint checks - DONE - [X] Split this out into multiple PRs - [X] SpaceAdapterFactory: !959 - [X] docstyle fixups: !960 - [ ] doc build fixups - [ ] devcontainer improvements (leaving the last two as this PR)
2023-02-27 08:42:22 +03:00
#CONDA_INFO_LEVEL ?= -q
Merged PR 572: Document experimental conda solver and enable Windows CI jobs This is a follow up to !557 (Make conda work on Windows) and addresses some of the lingering comments from that PR. - document the mamba slow solver workaround - use the mamba solver if available in Makefile - add windows jobs to the CI tasks as well
2022-08-18 07:43:22 +03:00
Merged PR 862: Add devcontainer support and other dev environment improvements This PR is about general dev environment improvements. Adds devcontainer support to help automatically prepare a dev environment inside either vscode of github codespaces. The container is build, tested, and published to an ACR in the MlosCore RG during the pipeline for easy reuse and quick dev env boot strapping and reproducible environments both locally and remotely. It also - Enables parallel Makefile builds - Adds additional helpful extensions to the vscode (e.g. copilot) - Moves some pip package installs to conda for better reproducibility - Fixes some issues regarding PYTHONPATH adjustment requirements for pytest and pylint. - And, since I noticed a config for it in vscode, I added pydocstyle checks, though they aren't configured for blocking atm since there's lots of complaints. Can decide whether to fix/force those later.
2023-02-08 00:43:01 +03:00
# Run make in parallel by default.
devcontainer build fixups for macOS clients (#874) # Pull Request ## Title Fixups to the devcontainer build for macOS clients. --- ## Description - Use a more portalable random number generator - Address some differences in docker.sock privileges. - Address differences in `stat` arguments. - Switch to wget mode for conda installation to workaround lack of arm64 apt repo. - When pulling base images to prime cache, use the appropriate architecture (for Windows we only support amd64 for now). - Remove `:latest` from `cache-from` args for `podman` compliance. - Address some differences in `sed` syntax. - Fixes #873 --- ## Type of Change - 🛠️ Bug fix --- ## Testing - local MacBook testing - CI testing for Linux --- ## Additional Notes Doesn't currently do builds for arm64 platform in the pipeline. Can work towards addressing that in the future.
2024-10-21 18:17:37 +03:00
MAKEFLAGS += -j$(shell nproc 2>/dev/null || sysctl -n hw.ncpu)
devcontainer style improvements (#358) * Disable line buffering * fail pylint checks on unused-import errors * make sure pycodestyle checks for indentation * increase code quality requirements * add some new extensions * bring pycodestyle checks inline with flake8 * add a prettier rc that instructs it to use our editorconfig --------- Co-authored-by: Sergiy Matusevych <sergiym@microsoft.com>
2023-05-16 01:19:20 +03:00
#MAKEFLAGS += -Oline
Merged PR 862: Add devcontainer support and other dev environment improvements This PR is about general dev environment improvements. Adds devcontainer support to help automatically prepare a dev environment inside either vscode of github codespaces. The container is build, tested, and published to an ACR in the MlosCore RG during the pipeline for easy reuse and quick dev env boot strapping and reproducible environments both locally and remotely. It also - Enables parallel Makefile builds - Adds additional helpful extensions to the vscode (e.g. copilot) - Moves some pip package installs to conda for better reproducibility - Fixes some issues regarding PYTHONPATH adjustment requirements for pytest and pylint. - And, since I noticed a config for it in vscode, I added pydocstyle checks, though they aren't configured for blocking atm since there's lots of complaints. Can decide whether to fix/force those later.
2023-02-08 00:43:01 +03:00
Merged PR 333: Initial MlosCore Implementation Adds basic `register` and `suggest` APIs for a couple of optimizer backends: `Skopt`, `Emukit`, `Random`. Includes example notebook for starters. Limited unit testing. Basic function documentation. Azure DevOps CI pipelines for `pylint` and `pytest` via the `mlos_core` `conda` `environment.yml`. Related work items: #274, #279
2022-03-14 22:37:52 +03:00
.PHONY: all
Staged Makefile Improvements (#762) - Introduce a new `make format` target that currently only calls `licenseheaders` but will call `isort` and `black` in the future - Introduce new variables to help make dependency tracking easier - Simplify `pytest` rules and allow running only a single module at a time. This PR is a precursor to pyproject.toml related build changes as well. --------- Co-authored-by: Sergiy Matusevych <sergiym@microsoft.com>
2024-07-01 18:57:17 +03:00
all: format check test dist dist-test doc | conda-env
Merged PR 333: Initial MlosCore Implementation Adds basic `register` and `suggest` APIs for a couple of optimizer backends: `Skopt`, `Emukit`, `Random`. Includes example notebook for starters. Limited unit testing. Basic function documentation. Azure DevOps CI pipelines for `pylint` and `pytest` via the `mlos_core` `conda` `environment.yml`. Related work items: #274, #279
2022-03-14 22:37:52 +03:00
.PHONY: conda-env
Merged PR 903: Combination of additional build and doc fixups This PR was originally aimed at just fixing the intermittent documentation build errors. That issue ultimately stems from conflicts between conda and pip installs of two different versions of `mistune` dependency. Along the way I found a handful of other improvements for build, pipeline, and documentation generation things. - Make sure we do the nightly build regardless of whether code has changed - Don't build the docs in parallel by default (trying to avoid an intermittent build failure) - Add basic support for testing devcontainer manually outside for vscode - Stop doing parallel builds so it's easier to debug, and move doc build/publishing to the devcontainer - Split windows conda env off, stop using gpy from conda-forge, and just build it locally with clang, gcc, or msvc - Adjust the devcontainer build so that more dependencies are built upfront during the container build. This makes the process more cacheable so that we can just pull the upstream version and not have to build as much locally which could delay first launch. - Import part of a fix from elsewhere - Switch to docker-compose style devcontainer that automatically also starts an nginx container for viewing the generated docs - Update some docs and doc building - [X] pydocstyle fixups - [X] Add some log filtering of the `make doc` output for easier debugging (e.g. don't care about missing examples for now) - [X] doc link lint checks - DONE - [X] Split this out into multiple PRs - [X] SpaceAdapterFactory: !959 - [X] docstyle fixups: !960 - [ ] doc build fixups - [ ] devcontainer improvements (leaving the last two as this PR)
2023-02-27 08:42:22 +03:00
conda-env: build/conda-env.${CONDA_ENV_NAME}.build-stamp
Merged PR 333: Initial MlosCore Implementation Adds basic `register` and `suggest` APIs for a couple of optimizer backends: `Skopt`, `Emukit`, `Random`. Includes example notebook for starters. Limited unit testing. Basic function documentation. Azure DevOps CI pipelines for `pylint` and `pytest` via the `mlos_core` `conda` `environment.yml`. Related work items: #274, #279
2022-03-14 22:37:52 +03:00
Introduce pyproject.toml build changes (#764) This PR builds off of #762 and #763 These are in part 1. followup fixups for #746 (e.g., to allow setuptools-scm to be pulled in at build time as a build dependency only when conda an pip have mismatched version issues), and 2. Modernization improvements to allow us to make better use of other tools (e.g., `black` that only accept `pyproject.toml` files as their configuration files). To do so, we move some configs from `setup.py` to `pyproject.toml` for each module. However, to retain the ability to rewrite URLs in published README.md files on PyPi as well as consistent version dependencies across modules without the need to manually specify version numbers (e.g., using `setuptools-scm`) we mark a few dependencies as dynamic and leave our existing logic inside the `setup.py` file. Finally, we reorganize the `version.py` file to be inside the module and fix a few previous omissions for `mlos_viz`.
2024-07-03 01:31:31 +03:00
MLOS_CORE_CONF_FILES := mlos_core/pyproject.toml mlos_core/setup.py mlos_core/MANIFEST.in
MLOS_BENCH_CONF_FILES := mlos_bench/pyproject.toml mlos_bench/setup.py mlos_bench/MANIFEST.in
MLOS_VIZ_CONF_FILES := mlos_viz/pyproject.toml mlos_viz/setup.py mlos_viz/MANIFEST.in
MLOS_GLOBAL_CONF_FILES := setup.cfg pyproject.toml
Staged Makefile Improvements (#762) - Introduce a new `make format` target that currently only calls `licenseheaders` but will call `isort` and `black` in the future - Introduce new variables to help make dependency tracking easier - Simplify `pytest` rules and allow running only a single module at a time. This PR is a precursor to pyproject.toml related build changes as well. --------- Co-authored-by: Sergiy Matusevych <sergiym@microsoft.com>
2024-07-01 18:57:17 +03:00
MLOS_PKGS := mlos_core mlos_bench mlos_viz
MLOS_PKG_CONF_FILES := $(MLOS_CORE_CONF_FILES) $(MLOS_BENCH_CONF_FILES) $(MLOS_VIZ_CONF_FILES) $(MLOS_GLOBAL_CONF_FILES)
build/conda-env.${CONDA_ENV_NAME}.build-stamp: ${ENV_YML} $(MLOS_PKG_CONF_FILES)
Merged PR 862: Add devcontainer support and other dev environment improvements This PR is about general dev environment improvements. Adds devcontainer support to help automatically prepare a dev environment inside either vscode of github codespaces. The container is build, tested, and published to an ACR in the MlosCore RG during the pipeline for easy reuse and quick dev env boot strapping and reproducible environments both locally and remotely. It also - Enables parallel Makefile builds - Adds additional helpful extensions to the vscode (e.g. copilot) - Moves some pip package installs to conda for better reproducibility - Fixes some issues regarding PYTHONPATH adjustment requirements for pytest and pylint. - And, since I noticed a config for it in vscode, I added pydocstyle checks, though they aren't configured for blocking atm since there's lots of complaints. Can decide whether to fix/force those later.
2023-02-08 00:43:01 +03:00
@echo "CONDA_SOLVER: ${CONDA_SOLVER}"
@echo "CONDA_EXPERIMENTAL_SOLVER: ${CONDA_EXPERIMENTAL_SOLVER}"
@echo "CONDA_INFO_LEVEL: ${CONDA_INFO_LEVEL}"
Merged PR 773: Fix pipelines for upstream changes to the recent miniconda release "experimental_solver" is now just known as "solver"
2022-12-06 22:54:00 +03:00
conda env list -q | grep -q "^${CONDA_ENV_NAME} " || conda env create ${CONDA_INFO_LEVEL} -n ${CONDA_ENV_NAME} -f ${ENV_YML}
conda env update ${CONDA_INFO_LEVEL} -n ${CONDA_ENV_NAME} --prune -f ${ENV_YML}
Staged Makefile Improvements (#762) - Introduce a new `make format` target that currently only calls `licenseheaders` but will call `isort` and `black` in the future - Introduce new variables to help make dependency tracking easier - Simplify `pytest` rules and allow running only a single module at a time. This PR is a precursor to pyproject.toml related build changes as well. --------- Co-authored-by: Sergiy Matusevych <sergiym@microsoft.com>
2024-07-01 18:57:17 +03:00
$(MAKE) clean-format clean-check clean-test clean-doc clean-dist
Merged PR 1026: Split test and check rules and improve test coverage publishing in pipelines and docs - make test and check targets for mlos_core and mlos_bench independent for faster execution and more clear output for `make test` and `make check` iteration when only hacking on one module locally - ensures that the full set still runs in the pipeline though - add support for skipping code coverage locally by setting `SKIP_COVERAGE=true` to run tests faster - optionally copy the code coverage data into our static html site for publishing (plus nice badges) - also, rather than racing and publishing the test results and code coverage for all of the different sub jobs, we only run and upload coverage via `pytest-azurepipelines` for the main one we're interested in to make reporting easier - also includes which job the test came from in the results - some other general `Makefile` cleanup Future work: - [X] Split the single pipeline into two pipelines and make the Windows one optional (since it's usually the one that takes the longest and we care a little bit less about it now that we have decontainer support) !1040
2023-03-16 22:47:01 +03:00
touch $@
Merged PR 333: Initial MlosCore Implementation Adds basic `register` and `suggest` APIs for a couple of optimizer backends: `Skopt`, `Emukit`, `Random`. Includes example notebook for starters. Limited unit testing. Basic function documentation. Azure DevOps CI pipelines for `pylint` and `pytest` via the `mlos_core` `conda` `environment.yml`. Related work items: #274, #279
2022-03-14 22:37:52 +03:00
Merged PR 557: Make conda work on Windows Related work items: #439
2022-08-12 19:43:26 +03:00
.PHONY: clean-conda-env
clean-conda-env:
Merged PR 903: Combination of additional build and doc fixups This PR was originally aimed at just fixing the intermittent documentation build errors. That issue ultimately stems from conflicts between conda and pip installs of two different versions of `mistune` dependency. Along the way I found a handful of other improvements for build, pipeline, and documentation generation things. - Make sure we do the nightly build regardless of whether code has changed - Don't build the docs in parallel by default (trying to avoid an intermittent build failure) - Add basic support for testing devcontainer manually outside for vscode - Stop doing parallel builds so it's easier to debug, and move doc build/publishing to the devcontainer - Split windows conda env off, stop using gpy from conda-forge, and just build it locally with clang, gcc, or msvc - Adjust the devcontainer build so that more dependencies are built upfront during the container build. This makes the process more cacheable so that we can just pull the upstream version and not have to build as much locally which could delay first launch. - Import part of a fix from elsewhere - Switch to docker-compose style devcontainer that automatically also starts an nginx container for viewing the generated docs - Update some docs and doc building - [X] pydocstyle fixups - [X] Add some log filtering of the `make doc` output for easier debugging (e.g. don't care about missing examples for now) - [X] doc link lint checks - DONE - [X] Split this out into multiple PRs - [X] SpaceAdapterFactory: !959 - [X] docstyle fixups: !960 - [ ] doc build fixups - [ ] devcontainer improvements (leaving the last two as this PR)
2023-02-27 08:42:22 +03:00
conda env remove -y ${CONDA_INFO_LEVEL} -n ${CONDA_ENV_NAME}
rm -f build/conda-env.${CONDA_ENV_NAME}.build-stamp
Merged PR 557: Make conda work on Windows Related work items: #439
2022-08-12 19:43:26 +03:00
Staged Makefile Improvements (#762) - Introduce a new `make format` target that currently only calls `licenseheaders` but will call `isort` and `black` in the future - Introduce new variables to help make dependency tracking easier - Simplify `pytest` rules and allow running only a single module at a time. This PR is a precursor to pyproject.toml related build changes as well. --------- Co-authored-by: Sergiy Matusevych <sergiym@microsoft.com>
2024-07-01 18:57:17 +03:00
# Here we make dynamic prereqs to apply to other targets that need to run in sequence.
FORMAT_PREREQS :=
.PHONY: format
format: build/format.${CONDA_ENV_NAME}.build-stamp
ifneq (,$(filter format,$(MAKECMDGOALS)))
FORMAT_PREREQS += build/format.${CONDA_ENV_NAME}.build-stamp
endif
Rework format lint checks for pre-commit (#904) # Pull Request ## Title Reworks build to use pre-commit for format and lint checks. --- ## Description As mentioned in #902 and #903, this refactors the build system to use [`pre-commit`](https://pre-commit.com) for the following reasons: - less complicated Makefile rules - pinned linter versions that can be checked with dependabot more explicitly for easier maintenance - See Also: #749 --- ## Type of Change - ✨ New feature - 🔄 Refactor - 📝 Documentation update - 🧪 Tests --- ## Testing Manually locally and usual CI tests. --- ## Additional Notes (optional) This builds off of #903 and is prep work to change the syntax of the codebase using `pyupgrade`. ---
2025-01-04 01:58:07 +03:00
FORMAT_COMMON_PREREQS := build/conda-env.${CONDA_ENV_NAME}.build-stamp
FORMAT_COMMON_PREREQS += .pre-commit-config.yaml
FORMAT_COMMON_PREREQS += $(MLOS_GLOBAL_CONF_FILES)
Prepare isort black formatters and checks (#766) Builds off of #762, #763, and #764. Prepares rules and configs to enable isort and black formatters and checks but doesn't enable them yet. After these are enabled (next PR) we will reformat all files and ignore that revision in git blame configs. Then, we can convert configs stored in `setup.cfg` and `.pylintrc` to the top level `pyproject.toml` and remove the older configs.
2024-07-03 23:57:49 +03:00
Remove use of yq (#908) # Pull Request ## Title Remove use of yq. --- ## Description Removes the use of yq which doesn't always exist and running it via conda apparently has issues. --- ## Type of Change - 🛠️ Bug fix --- ## Testing CI --- ## Additional Notes (optional) Fixes a bug introduced in #904 ---
2025-01-04 03:28:32 +03:00
# Formatting pre-commit hooks are marked with the "manual" stage in .pre-commit-config.yaml
# Since yq is not installed everywhere we need to sync the list here as well.
FORMATTERS := end-of-file-fixer pretty-format-json trailingwhitespace licenseheaders pyupgrade isort black docformatter
Prepare isort black formatters and checks (#766) Builds off of #762, #763, and #764. Prepares rules and configs to enable isort and black formatters and checks but doesn't enable them yet. After these are enabled (next PR) we will reformat all files and ignore that revision in git blame configs. Then, we can convert configs stored in `setup.cfg` and `.pylintrc` to the top level `pyproject.toml` and remove the older configs.
2024-07-03 23:57:49 +03:00
Rework format lint checks for pre-commit (#904) # Pull Request ## Title Reworks build to use pre-commit for format and lint checks. --- ## Description As mentioned in #902 and #903, this refactors the build system to use [`pre-commit`](https://pre-commit.com) for the following reasons: - less complicated Makefile rules - pinned linter versions that can be checked with dependabot more explicitly for easier maintenance - See Also: #749 --- ## Type of Change - ✨ New feature - 🔄 Refactor - 📝 Documentation update - 🧪 Tests --- ## Testing Manually locally and usual CI tests. --- ## Additional Notes (optional) This builds off of #903 and is prep work to change the syntax of the codebase using `pyupgrade`. ---
2025-01-04 01:58:07 +03:00
# Provide convenience methods to call individual formatters and checkers via `make` as well.
# e.g., `make black` or `make pylint`
.PHONY: $(FORMATTERS) $(CHECKERS)
.NOTPARALLEL: $(FORMATTERS)
$(FORMATTERS) $(CHECKERS): $(MLOS_CORE_PYTHON_FILES)
$(FORMATTERS) $(CHECKERS): $(MLOS_BENCH_PYTHON_FILES)
$(FORMATTERS) $(CHECKERS): $(MLOS_VIZ_PYTHON_FILES)
$(FORMATTERS) $(CHECKERS): $(FORMAT_COMMON_PREREQS)
conda run -n ${CONDA_ENV_NAME} pre-commit run -v --all-files $@
Prepare isort black formatters and checks (#766) Builds off of #762, #763, and #764. Prepares rules and configs to enable isort and black formatters and checks but doesn't enable them yet. After these are enabled (next PR) we will reformat all files and ignore that revision in git blame configs. Then, we can convert configs stored in `setup.cfg` and `.pylintrc` to the top level `pyproject.toml` and remove the older configs.
2024-07-03 23:57:49 +03:00
Rework format lint checks for pre-commit (#904) # Pull Request ## Title Reworks build to use pre-commit for format and lint checks. --- ## Description As mentioned in #902 and #903, this refactors the build system to use [`pre-commit`](https://pre-commit.com) for the following reasons: - less complicated Makefile rules - pinned linter versions that can be checked with dependabot more explicitly for easier maintenance - See Also: #749 --- ## Type of Change - ✨ New feature - 🔄 Refactor - 📝 Documentation update - 🧪 Tests --- ## Testing Manually locally and usual CI tests. --- ## Additional Notes (optional) This builds off of #903 and is prep work to change the syntax of the codebase using `pyupgrade`. ---
2025-01-04 01:58:07 +03:00
build/format.${CONDA_ENV_NAME}.build-stamp:
conda run -n ${CONDA_ENV_NAME} pre-commit run -v --all-files --hook-stage manual
Enable black, isort, and doc formatters and checks (#774) Follow on work to #766. This enabled both formatters and applies their changes to the repo. Additionally, since `black` does not make changes to comments nor docstrings, we also enable `docformatter` to reformat docstrings which better aligns with `pydocstyle` rules as well. Without this additional change (and some manual fixups), `pycodestyle` and `pylint` would still complain about line lengths, for instance. Finally, we make a minor adjustment to the max line length setting it to 99 (which is also accepted and mentioned in pep8) instead of 88 to avoid some comment (especially linter overrides) wrapping.
2024-07-12 22:56:14 +03:00
touch $@
Merged PR 333: Initial MlosCore Implementation Adds basic `register` and `suggest` APIs for a couple of optimizer backends: `Skopt`, `Emukit`, `Random`. Includes example notebook for starters. Limited unit testing. Basic function documentation. Azure DevOps CI pipelines for `pylint` and `pytest` via the `mlos_core` `conda` `environment.yml`. Related work items: #274, #279
2022-03-14 22:37:52 +03:00
.PHONY: check
Rework format lint checks for pre-commit (#904) # Pull Request ## Title Reworks build to use pre-commit for format and lint checks. --- ## Description As mentioned in #902 and #903, this refactors the build system to use [`pre-commit`](https://pre-commit.com) for the following reasons: - less complicated Makefile rules - pinned linter versions that can be checked with dependabot more explicitly for easier maintenance - See Also: #749 --- ## Type of Change - ✨ New feature - 🔄 Refactor - 📝 Documentation update - 🧪 Tests --- ## Testing Manually locally and usual CI tests. --- ## Additional Notes (optional) This builds off of #903 and is prep work to change the syntax of the codebase using `pyupgrade`. ---
2025-01-04 01:58:07 +03:00
check: build/check.${CONDA_ENV_NAME}.build-stamp
Enable black, isort, and doc formatters and checks (#774) Follow on work to #766. This enabled both formatters and applies their changes to the repo. Additionally, since `black` does not make changes to comments nor docstrings, we also enable `docformatter` to reformat docstrings which better aligns with `pydocstyle` rules as well. Without this additional change (and some manual fixups), `pycodestyle` and `pylint` would still complain about line lengths, for instance. Finally, we make a minor adjustment to the max line length setting it to 99 (which is also accepted and mentioned in pep8) instead of 88 to avoid some comment (especially linter overrides) wrapping.
2024-07-12 22:56:14 +03:00
Rework format lint checks for pre-commit (#904) # Pull Request ## Title Reworks build to use pre-commit for format and lint checks. --- ## Description As mentioned in #902 and #903, this refactors the build system to use [`pre-commit`](https://pre-commit.com) for the following reasons: - less complicated Makefile rules - pinned linter versions that can be checked with dependabot more explicitly for easier maintenance - See Also: #749 --- ## Type of Change - ✨ New feature - 🔄 Refactor - 📝 Documentation update - 🧪 Tests --- ## Testing Manually locally and usual CI tests. --- ## Additional Notes (optional) This builds off of #903 and is prep work to change the syntax of the codebase using `pyupgrade`. ---
2025-01-04 01:58:07 +03:00
CHECK_COMMON_PREREQS := build/conda-env.${CONDA_ENV_NAME}.build-stamp
CHECK_COMMON_PREREQS += .pre-commit-config.yaml
CHECK_COMMON_PREREQS += $(FORMAT_PREREQS)
CHECK_COMMON_PREREQS += $(MLOS_GLOBAL_CONF_FILES)
Prepare isort black formatters and checks (#766) Builds off of #762, #763, and #764. Prepares rules and configs to enable isort and black formatters and checks but doesn't enable them yet. After these are enabled (next PR) we will reformat all files and ignore that revision in git blame configs. Then, we can convert configs stored in `setup.cfg` and `.pylintrc` to the top level `pyproject.toml` and remove the older configs.
2024-07-03 23:57:49 +03:00
Rework format lint checks for pre-commit (#904) # Pull Request ## Title Reworks build to use pre-commit for format and lint checks. --- ## Description As mentioned in #902 and #903, this refactors the build system to use [`pre-commit`](https://pre-commit.com) for the following reasons: - less complicated Makefile rules - pinned linter versions that can be checked with dependabot more explicitly for easier maintenance - See Also: #749 --- ## Type of Change - ✨ New feature - 🔄 Refactor - 📝 Documentation update - 🧪 Tests --- ## Testing Manually locally and usual CI tests. --- ## Additional Notes (optional) This builds off of #903 and is prep work to change the syntax of the codebase using `pyupgrade`. ---
2025-01-04 01:58:07 +03:00
build/check.${CONDA_ENV_NAME}.build-stamp: $(MLOS_CORE_PYTHON_FILES)
build/check.${CONDA_ENV_NAME}.build-stamp: $(MLOS_BENCH_PYTHON_FILES)
build/check.${CONDA_ENV_NAME}.build-stamp: $(MLOS_VIZ_PYTHON_FILES)
build/check.${CONDA_ENV_NAME}.build-stamp: $(CHECK_COMMON_PREREQS)
SKIP=`echo '$(FORMATTERS)' | tr ' ' ','` conda run -n ${CONDA_ENV_NAME} pre-commit run -v --all-files
Merged PR 1054: Prepwork for OSSing - include some boilerplate from the original MLOS repo - adding license headers on scripts - add rules for checking licenseheaders
2023-03-18 00:20:39 +03:00
touch $@
Merged PR 903: Combination of additional build and doc fixups This PR was originally aimed at just fixing the intermittent documentation build errors. That issue ultimately stems from conflicts between conda and pip installs of two different versions of `mistune` dependency. Along the way I found a handful of other improvements for build, pipeline, and documentation generation things. - Make sure we do the nightly build regardless of whether code has changed - Don't build the docs in parallel by default (trying to avoid an intermittent build failure) - Add basic support for testing devcontainer manually outside for vscode - Stop doing parallel builds so it's easier to debug, and move doc build/publishing to the devcontainer - Split windows conda env off, stop using gpy from conda-forge, and just build it locally with clang, gcc, or msvc - Adjust the devcontainer build so that more dependencies are built upfront during the container build. This makes the process more cacheable so that we can just pull the upstream version and not have to build as much locally which could delay first launch. - Import part of a fix from elsewhere - Switch to docker-compose style devcontainer that automatically also starts an nginx container for viewing the generated docs - Update some docs and doc building - [X] pydocstyle fixups - [X] Add some log filtering of the `make doc` output for easier debugging (e.g. don't care about missing examples for now) - [X] doc link lint checks - DONE - [X] Split this out into multiple PRs - [X] SpaceAdapterFactory: !959 - [X] docstyle fixups: !960 - [ ] doc build fixups - [ ] devcontainer improvements (leaving the last two as this PR)
2023-02-27 08:42:22 +03:00
.PHONY: cspell
ifeq ($(DOCKER),)
cspell:
@echo "NOTE: docker is not available. Skipping cspell check."
else
cspell: build/cspell-container.build-stamp
./.devcontainer/scripts/run-cspell.sh
endif
Staged Makefile Improvements (#762) - Introduce a new `make format` target that currently only calls `licenseheaders` but will call `isort` and `black` in the future - Introduce new variables to help make dependency tracking easier - Simplify `pytest` rules and allow running only a single module at a time. This PR is a precursor to pyproject.toml related build changes as well. --------- Co-authored-by: Sergiy Matusevych <sergiym@microsoft.com>
2024-07-01 18:57:17 +03:00
build/cspell-container.build-stamp: $(FORMAT_PREREQS)
Merged PR 903: Combination of additional build and doc fixups This PR was originally aimed at just fixing the intermittent documentation build errors. That issue ultimately stems from conflicts between conda and pip installs of two different versions of `mistune` dependency. Along the way I found a handful of other improvements for build, pipeline, and documentation generation things. - Make sure we do the nightly build regardless of whether code has changed - Don't build the docs in parallel by default (trying to avoid an intermittent build failure) - Add basic support for testing devcontainer manually outside for vscode - Stop doing parallel builds so it's easier to debug, and move doc build/publishing to the devcontainer - Split windows conda env off, stop using gpy from conda-forge, and just build it locally with clang, gcc, or msvc - Adjust the devcontainer build so that more dependencies are built upfront during the container build. This makes the process more cacheable so that we can just pull the upstream version and not have to build as much locally which could delay first launch. - Import part of a fix from elsewhere - Switch to docker-compose style devcontainer that automatically also starts an nginx container for viewing the generated docs - Update some docs and doc building - [X] pydocstyle fixups - [X] Add some log filtering of the `make doc` output for easier debugging (e.g. don't care about missing examples for now) - [X] doc link lint checks - DONE - [X] Split this out into multiple PRs - [X] SpaceAdapterFactory: !959 - [X] docstyle fixups: !960 - [ ] doc build fixups - [ ] devcontainer improvements (leaving the last two as this PR)
2023-02-27 08:42:22 +03:00
# Build the docker image with cspell in it.
$(MAKE) -C .devcontainer/build cspell
Merged PR 1026: Split test and check rules and improve test coverage publishing in pipelines and docs - make test and check targets for mlos_core and mlos_bench independent for faster execution and more clear output for `make test` and `make check` iteration when only hacking on one module locally - ensures that the full set still runs in the pipeline though - add support for skipping code coverage locally by setting `SKIP_COVERAGE=true` to run tests faster - optionally copy the code coverage data into our static html site for publishing (plus nice badges) - also, rather than racing and publishing the test results and code coverage for all of the different sub jobs, we only run and upload coverage via `pytest-azurepipelines` for the main one we're interested in to make reporting easier - also includes which job the test came from in the results - some other general `Makefile` cleanup Future work: - [X] Split the single pipeline into two pipelines and make the Windows one optional (since it's usually the one that takes the longest and we care a little bit less about it now that we have decontainer support) !1040
2023-03-16 22:47:01 +03:00
touch $@
Merged PR 333: Initial MlosCore Implementation Adds basic `register` and `suggest` APIs for a couple of optimizer backends: `Skopt`, `Emukit`, `Random`. Includes example notebook for starters. Limited unit testing. Basic function documentation. Azure DevOps CI pipelines for `pylint` and `pytest` via the `mlos_core` `conda` `environment.yml`. Related work items: #274, #279
2022-03-14 22:37:52 +03:00
Switch to Github Actions for CI/CD runs (#274)
2023-03-18 23:52:26 +03:00
.PHONY: markdown-link-check
ifeq ($(DOCKER),)
markdown-link-check:
@echo "NOTE: docker is not available. Skipping markdown-link-check check."
else
markdown-link-check: build/markdown-link-check-container.build-stamp
./.devcontainer/scripts/run-markdown-link-check.sh
endif
Staged Makefile Improvements (#762) - Introduce a new `make format` target that currently only calls `licenseheaders` but will call `isort` and `black` in the future - Introduce new variables to help make dependency tracking easier - Simplify `pytest` rules and allow running only a single module at a time. This PR is a precursor to pyproject.toml related build changes as well. --------- Co-authored-by: Sergiy Matusevych <sergiym@microsoft.com>
2024-07-01 18:57:17 +03:00
build/markdown-link-check-container.build-stamp: $(FORMAT_PREREQS)
Switch to Github Actions for CI/CD runs (#274)
2023-03-18 23:52:26 +03:00
# Build the docker image with markdown-link-check in it.
$(MAKE) -C .devcontainer/build markdown-link-check
touch $@
Merged PR 333: Initial MlosCore Implementation Adds basic `register` and `suggest` APIs for a couple of optimizer backends: `Skopt`, `Emukit`, `Random`. Includes example notebook for starters. Limited unit testing. Basic function documentation. Azure DevOps CI pipelines for `pylint` and `pytest` via the `mlos_core` `conda` `environment.yml`. Related work items: #274, #279
2022-03-14 22:37:52 +03:00
.PHONY: test
Fixups to example notebook for new APIs (#834) - [x] enable notebook execution in CI pipelines to prevent these mistakes in the future --------- Co-authored-by: Sergiy Matusevych <sergiym@microsoft.com>
2024-08-08 02:20:17 +03:00
test: pytest notebook-exec-test
Merged PR 333: Initial MlosCore Implementation Adds basic `register` and `suggest` APIs for a couple of optimizer backends: `Skopt`, `Emukit`, `Random`. Includes example notebook for starters. Limited unit testing. Basic function documentation. Azure DevOps CI pipelines for `pylint` and `pytest` via the `mlos_core` `conda` `environment.yml`. Related work items: #274, #279
2022-03-14 22:37:52 +03:00
Staged Makefile Improvements (#762) - Introduce a new `make format` target that currently only calls `licenseheaders` but will call `isort` and `black` in the future - Introduce new variables to help make dependency tracking easier - Simplify `pytest` rules and allow running only a single module at a time. This PR is a precursor to pyproject.toml related build changes as well. --------- Co-authored-by: Sergiy Matusevych <sergiym@microsoft.com>
2024-07-01 18:57:17 +03:00
PYTEST_CONF_FILES := $(MLOS_GLOBAL_CONF_FILES) conftest.py
Merged PR 1026: Split test and check rules and improve test coverage publishing in pipelines and docs - make test and check targets for mlos_core and mlos_bench independent for faster execution and more clear output for `make test` and `make check` iteration when only hacking on one module locally - ensures that the full set still runs in the pipeline though - add support for skipping code coverage locally by setting `SKIP_COVERAGE=true` to run tests faster - optionally copy the code coverage data into our static html site for publishing (plus nice badges) - also, rather than racing and publishing the test results and code coverage for all of the different sub jobs, we only run and upload coverage via `pytest-azurepipelines` for the main one we're interested in to make reporting easier - also includes which job the test came from in the results - some other general `Makefile` cleanup Future work: - [X] Split the single pipeline into two pipelines and make the Windows one optional (since it's usually the one that takes the longest and we care a little bit less about it now that we have decontainer support) !1040
2023-03-16 22:47:01 +03:00
Merged PR 333: Initial MlosCore Implementation Adds basic `register` and `suggest` APIs for a couple of optimizer backends: `Skopt`, `Emukit`, `Random`. Includes example notebook for starters. Limited unit testing. Basic function documentation. Azure DevOps CI pipelines for `pylint` and `pytest` via the `mlos_core` `conda` `environment.yml`. Related work items: #274, #279
2022-03-14 22:37:52 +03:00
.PHONY: pytest
Merged PR 903: Combination of additional build and doc fixups This PR was originally aimed at just fixing the intermittent documentation build errors. That issue ultimately stems from conflicts between conda and pip installs of two different versions of `mistune` dependency. Along the way I found a handful of other improvements for build, pipeline, and documentation generation things. - Make sure we do the nightly build regardless of whether code has changed - Don't build the docs in parallel by default (trying to avoid an intermittent build failure) - Add basic support for testing devcontainer manually outside for vscode - Stop doing parallel builds so it's easier to debug, and move doc build/publishing to the devcontainer - Split windows conda env off, stop using gpy from conda-forge, and just build it locally with clang, gcc, or msvc - Adjust the devcontainer build so that more dependencies are built upfront during the container build. This makes the process more cacheable so that we can just pull the upstream version and not have to build as much locally which could delay first launch. - Import part of a fix from elsewhere - Switch to docker-compose style devcontainer that automatically also starts an nginx container for viewing the generated docs - Update some docs and doc building - [X] pydocstyle fixups - [X] Add some log filtering of the `make doc` output for easier debugging (e.g. don't care about missing examples for now) - [X] doc link lint checks - DONE - [X] Split this out into multiple PRs - [X] SpaceAdapterFactory: !959 - [X] docstyle fixups: !960 - [ ] doc build fixups - [ ] devcontainer improvements (leaving the last two as this PR)
2023-02-27 08:42:22 +03:00
pytest: conda-env build/pytest.${CONDA_ENV_NAME}.build-stamp
Merged PR 333: Initial MlosCore Implementation Adds basic `register` and `suggest` APIs for a couple of optimizer backends: `Skopt`, `Emukit`, `Random`. Includes example notebook for starters. Limited unit testing. Basic function documentation. Azure DevOps CI pipelines for `pylint` and `pytest` via the `mlos_core` `conda` `environment.yml`. Related work items: #274, #279
2022-03-14 22:37:52 +03:00
Staged Makefile Improvements (#762) - Introduce a new `make format` target that currently only calls `licenseheaders` but will call `isort` and `black` in the future - Introduce new variables to help make dependency tracking easier - Simplify `pytest` rules and allow running only a single module at a time. This PR is a precursor to pyproject.toml related build changes as well. --------- Co-authored-by: Sergiy Matusevych <sergiym@microsoft.com>
2024-07-01 18:57:17 +03:00
pytest-mlos-core: build/pytest.mlos_core.${CONDA_ENV_NAME}.needs-build-stamp
pytest-mlos-bench: build/pytest.mlos_bench.${CONDA_ENV_NAME}.needs-build-stamp
pytest-mlos-viz: build/pytest.mlos_viz.${CONDA_ENV_NAME}.needs-build-stamp
Merged PR 1026: Split test and check rules and improve test coverage publishing in pipelines and docs - make test and check targets for mlos_core and mlos_bench independent for faster execution and more clear output for `make test` and `make check` iteration when only hacking on one module locally - ensures that the full set still runs in the pipeline though - add support for skipping code coverage locally by setting `SKIP_COVERAGE=true` to run tests faster - optionally copy the code coverage data into our static html site for publishing (plus nice badges) - also, rather than racing and publishing the test results and code coverage for all of the different sub jobs, we only run and upload coverage via `pytest-azurepipelines` for the main one we're interested in to make reporting easier - also includes which job the test came from in the results - some other general `Makefile` cleanup Future work: - [X] Split the single pipeline into two pipelines and make the Windows one optional (since it's usually the one that takes the longest and we care a little bit less about it now that we have decontainer support) !1040
2023-03-16 22:47:01 +03:00
Staged Makefile Improvements (#762) - Introduce a new `make format` target that currently only calls `licenseheaders` but will call `isort` and `black` in the future - Introduce new variables to help make dependency tracking easier - Simplify `pytest` rules and allow running only a single module at a time. This PR is a precursor to pyproject.toml related build changes as well. --------- Co-authored-by: Sergiy Matusevych <sergiym@microsoft.com>
2024-07-01 18:57:17 +03:00
build/pytest.mlos_core.${CONDA_ENV_NAME}.needs-build-stamp: $(MLOS_CORE_PYTHON_FILES) $(MLOS_CORE_CONF_FILES)
build/pytest.mlos_core.${CONDA_ENV_NAME}.needs-build-stamp: PYTEST_MODULE := mlos_core
build/pytest.mlos_bench.${CONDA_ENV_NAME}.needs-build-stamp: $(MLOS_BENCH_PYTHON_FILES) $(MLOS_BENCH_CONF_FILES)
build/pytest.mlos_bench.${CONDA_ENV_NAME}.needs-build-stamp: PYTEST_MODULE := mlos_bench
Merged PR 1026: Split test and check rules and improve test coverage publishing in pipelines and docs - make test and check targets for mlos_core and mlos_bench independent for faster execution and more clear output for `make test` and `make check` iteration when only hacking on one module locally - ensures that the full set still runs in the pipeline though - add support for skipping code coverage locally by setting `SKIP_COVERAGE=true` to run tests faster - optionally copy the code coverage data into our static html site for publishing (plus nice badges) - also, rather than racing and publishing the test results and code coverage for all of the different sub jobs, we only run and upload coverage via `pytest-azurepipelines` for the main one we're interested in to make reporting easier - also includes which job the test came from in the results - some other general `Makefile` cleanup Future work: - [X] Split the single pipeline into two pipelines and make the Windows one optional (since it's usually the one that takes the longest and we care a little bit less about it now that we have decontainer support) !1040
2023-03-16 22:47:01 +03:00
Staged Makefile Improvements (#762) - Introduce a new `make format` target that currently only calls `licenseheaders` but will call `isort` and `black` in the future - Introduce new variables to help make dependency tracking easier - Simplify `pytest` rules and allow running only a single module at a time. This PR is a precursor to pyproject.toml related build changes as well. --------- Co-authored-by: Sergiy Matusevych <sergiym@microsoft.com>
2024-07-01 18:57:17 +03:00
build/pytest.mlos_viz.${CONDA_ENV_NAME}.needs-build-stamp: $(MLOS_VIZ_PYTHON_FILES) $(MLOS_VIZ_CONF_FILES)
build/pytest.mlos_viz.${CONDA_ENV_NAME}.needs-build-stamp: PYTEST_MODULE := mlos_viz
Rework format lint checks for pre-commit (#904) # Pull Request ## Title Reworks build to use pre-commit for format and lint checks. --- ## Description As mentioned in #902 and #903, this refactors the build system to use [`pre-commit`](https://pre-commit.com) for the following reasons: - less complicated Makefile rules - pinned linter versions that can be checked with dependabot more explicitly for easier maintenance - See Also: #749 --- ## Type of Change - ✨ New feature - 🔄 Refactor - 📝 Documentation update - 🧪 Tests --- ## Testing Manually locally and usual CI tests. --- ## Additional Notes (optional) This builds off of #903 and is prep work to change the syntax of the codebase using `pyupgrade`. ---
2025-01-04 01:58:07 +03:00
# Individual package test rules (for tight loop dev work).
Staged Makefile Improvements (#762) - Introduce a new `make format` target that currently only calls `licenseheaders` but will call `isort` and `black` in the future - Introduce new variables to help make dependency tracking easier - Simplify `pytest` rules and allow running only a single module at a time. This PR is a precursor to pyproject.toml related build changes as well. --------- Co-authored-by: Sergiy Matusevych <sergiym@microsoft.com>
2024-07-01 18:57:17 +03:00
# Skip code coverage tests for these.
PYTEST_COMMON_PREREQS := build/conda-env.${CONDA_ENV_NAME}.build-stamp
PYTEST_COMMON_PREREQS += $(FORMAT_PREREQS)
PYTEST_COMMON_PREREQS += $(PYTEST_CONF_FILES)
build/pytest.%.${CONDA_ENV_NAME}.needs-build-stamp: $(PYTEST_COMMON_PREREQS)
conda run -n ${CONDA_ENV_NAME} pytest $(PYTEST_EXTRA_OPTIONS) $(PYTEST_MODULE)
Introduce basic experiment visualization module mlos_viz via dabl (#624) Adds a basic `mlos_viz.plot(exp)` style API for simple visualizations of `ExperimentData` results relative to the experiment's objectives (building off of #628 and https://github.com/dabl/dabl/pull/335). Note: this PR currently omits unit tests for the new module due to the complexity of testing visualizations. We intend to add this in future PRs. There is however, a working example of its use here right now: https://github.com/Microsoft-CISL/sqlite-autotuning/pull/41 --------- Co-authored-by: Sergiy Matusevych <sergiy.matusevych@gmail.com>
2024-01-16 21:48:20 +03:00
touch $@
Merged PR 1026: Split test and check rules and improve test coverage publishing in pipelines and docs - make test and check targets for mlos_core and mlos_bench independent for faster execution and more clear output for `make test` and `make check` iteration when only hacking on one module locally - ensures that the full set still runs in the pipeline though - add support for skipping code coverage locally by setting `SKIP_COVERAGE=true` to run tests faster - optionally copy the code coverage data into our static html site for publishing (plus nice badges) - also, rather than racing and publishing the test results and code coverage for all of the different sub jobs, we only run and upload coverage via `pytest-azurepipelines` for the main one we're interested in to make reporting easier - also includes which job the test came from in the results - some other general `Makefile` cleanup Future work: - [X] Split the single pipeline into two pipelines and make the Windows one optional (since it's usually the one that takes the longest and we care a little bit less about it now that we have decontainer support) !1040
2023-03-16 22:47:01 +03:00
PYTEST_OPTIONS :=
# Allow optionally skipping coverage calculations during local testing to skip up inner dev loop.
SKIP_COVERAGE := $(shell echo $${SKIP_COVERAGE:-} | grep -i -x -e 1 -e true)
ifeq ($(SKIP_COVERAGE),)
Staged Makefile Improvements (#762) - Introduce a new `make format` target that currently only calls `licenseheaders` but will call `isort` and `black` in the future - Introduce new variables to help make dependency tracking easier - Simplify `pytest` rules and allow running only a single module at a time. This PR is a precursor to pyproject.toml related build changes as well. --------- Co-authored-by: Sergiy Matusevych <sergiym@microsoft.com>
2024-07-01 18:57:17 +03:00
PYTEST_OPTIONS += --cov=. --cov-append --cov-fail-under=92 --cov-report=xml --cov-report=html --junitxml=junit/test-results.xml --local-badge-output-dir=doc/source/badges/
Merged PR 1026: Split test and check rules and improve test coverage publishing in pipelines and docs - make test and check targets for mlos_core and mlos_bench independent for faster execution and more clear output for `make test` and `make check` iteration when only hacking on one module locally - ensures that the full set still runs in the pipeline though - add support for skipping code coverage locally by setting `SKIP_COVERAGE=true` to run tests faster - optionally copy the code coverage data into our static html site for publishing (plus nice badges) - also, rather than racing and publishing the test results and code coverage for all of the different sub jobs, we only run and upload coverage via `pytest-azurepipelines` for the main one we're interested in to make reporting easier - also includes which job the test came from in the results - some other general `Makefile` cleanup Future work: - [X] Split the single pipeline into two pipelines and make the Windows one optional (since it's usually the one that takes the longest and we care a little bit less about it now that we have decontainer support) !1040
2023-03-16 22:47:01 +03:00
endif
Staged Makefile Improvements (#762) - Introduce a new `make format` target that currently only calls `licenseheaders` but will call `isort` and `black` in the future - Introduce new variables to help make dependency tracking easier - Simplify `pytest` rules and allow running only a single module at a time. This PR is a precursor to pyproject.toml related build changes as well. --------- Co-authored-by: Sergiy Matusevych <sergiym@microsoft.com>
2024-07-01 18:57:17 +03:00
# Global pytest rule that also produces code coverage for the pipeline.
Merged PR 1026: Split test and check rules and improve test coverage publishing in pipelines and docs - make test and check targets for mlos_core and mlos_bench independent for faster execution and more clear output for `make test` and `make check` iteration when only hacking on one module locally - ensures that the full set still runs in the pipeline though - add support for skipping code coverage locally by setting `SKIP_COVERAGE=true` to run tests faster - optionally copy the code coverage data into our static html site for publishing (plus nice badges) - also, rather than racing and publishing the test results and code coverage for all of the different sub jobs, we only run and upload coverage via `pytest-azurepipelines` for the main one we're interested in to make reporting easier - also includes which job the test came from in the results - some other general `Makefile` cleanup Future work: - [X] Split the single pipeline into two pipelines and make the Windows one optional (since it's usually the one that takes the longest and we care a little bit less about it now that we have decontainer support) !1040
2023-03-16 22:47:01 +03:00
# NOTE: When run locally, the junit/test-results.xml will only include the
# tests from the latest run, but this file is only used for upstream reporting,
# so probably shouldn't matter.
Staged Makefile Improvements (#762) - Introduce a new `make format` target that currently only calls `licenseheaders` but will call `isort` and `black` in the future - Introduce new variables to help make dependency tracking easier - Simplify `pytest` rules and allow running only a single module at a time. This PR is a precursor to pyproject.toml related build changes as well. --------- Co-authored-by: Sergiy Matusevych <sergiym@microsoft.com>
2024-07-01 18:57:17 +03:00
build/pytest.${CONDA_ENV_NAME}.build-stamp: $(PYTEST_COMMON_PREREQS)
build/pytest.${CONDA_ENV_NAME}.build-stamp: $(MLOS_CORE_PYTHON_FILES) $(MLOS_CORE_CONF_FILES)
build/pytest.${CONDA_ENV_NAME}.build-stamp: $(MLOS_BENCH_PYTHON_FILES) $(MLOS_BENCH_CONF_FILES)
build/pytest.${CONDA_ENV_NAME}.build-stamp: $(MLOS_VIZ_PYTHON_FILES) $(MLOS_VIZ_CONF_FILES)
Introduce basic experiment visualization module mlos_viz via dabl (#624) Adds a basic `mlos_viz.plot(exp)` style API for simple visualizations of `ExperimentData` results relative to the experiment's objectives (building off of #628 and https://github.com/dabl/dabl/pull/335). Note: this PR currently omits unit tests for the new module due to the complexity of testing visualizations. We intend to add this in future PRs. There is however, a working example of its use here right now: https://github.com/Microsoft-CISL/sqlite-autotuning/pull/41 --------- Co-authored-by: Sergiy Matusevych <sergiy.matusevych@gmail.com>
2024-01-16 21:48:20 +03:00
build/pytest.${CONDA_ENV_NAME}.build-stamp:
Staged Makefile Improvements (#762) - Introduce a new `make format` target that currently only calls `licenseheaders` but will call `isort` and `black` in the future - Introduce new variables to help make dependency tracking easier - Simplify `pytest` rules and allow running only a single module at a time. This PR is a precursor to pyproject.toml related build changes as well. --------- Co-authored-by: Sergiy Matusevych <sergiym@microsoft.com>
2024-07-01 18:57:17 +03:00
# Remove the markers for individual targets (above).
for pytest_module in $(MLOS_PKGS); do rm -f build/pytest.$${pytest_module}.${CONDA_ENV_NAME}.build-stamp; done
# Run pytest for the modules: $(MLOS_PKGS)
Merged PR 1026: Split test and check rules and improve test coverage publishing in pipelines and docs - make test and check targets for mlos_core and mlos_bench independent for faster execution and more clear output for `make test` and `make check` iteration when only hacking on one module locally - ensures that the full set still runs in the pipeline though - add support for skipping code coverage locally by setting `SKIP_COVERAGE=true` to run tests faster - optionally copy the code coverage data into our static html site for publishing (plus nice badges) - also, rather than racing and publishing the test results and code coverage for all of the different sub jobs, we only run and upload coverage via `pytest-azurepipelines` for the main one we're interested in to make reporting easier - also includes which job the test came from in the results - some other general `Makefile` cleanup Future work: - [X] Split the single pipeline into two pipelines and make the Windows one optional (since it's usually the one that takes the longest and we care a little bit less about it now that we have decontainer support) !1040
2023-03-16 22:47:01 +03:00
mkdir -p doc/source/badges/
Staged Makefile Improvements (#762) - Introduce a new `make format` target that currently only calls `licenseheaders` but will call `isort` and `black` in the future - Introduce new variables to help make dependency tracking easier - Simplify `pytest` rules and allow running only a single module at a time. This PR is a precursor to pyproject.toml related build changes as well. --------- Co-authored-by: Sergiy Matusevych <sergiym@microsoft.com>
2024-07-01 18:57:17 +03:00
conda run -n ${CONDA_ENV_NAME} pytest $(PYTEST_OPTIONS) $(PYTEST_EXTRA_OPTIONS) $(MLOS_PKGS)
# Global success. Mark the individual targets as done again.
for pytest_module in $(MLOS_PKGS); do touch build/pytest.$${pytest_module}.${CONDA_ENV_NAME}.build-stamp; done
Merged PR 1026: Split test and check rules and improve test coverage publishing in pipelines and docs - make test and check targets for mlos_core and mlos_bench independent for faster execution and more clear output for `make test` and `make check` iteration when only hacking on one module locally - ensures that the full set still runs in the pipeline though - add support for skipping code coverage locally by setting `SKIP_COVERAGE=true` to run tests faster - optionally copy the code coverage data into our static html site for publishing (plus nice badges) - also, rather than racing and publishing the test results and code coverage for all of the different sub jobs, we only run and upload coverage via `pytest-azurepipelines` for the main one we're interested in to make reporting easier - also includes which job the test came from in the results - some other general `Makefile` cleanup Future work: - [X] Split the single pipeline into two pipelines and make the Windows one optional (since it's usually the one that takes the longest and we care a little bit less about it now that we have decontainer support) !1040
2023-03-16 22:47:01 +03:00
touch $@
Merged PR 333: Initial MlosCore Implementation Adds basic `register` and `suggest` APIs for a couple of optimizer backends: `Skopt`, `Emukit`, `Random`. Includes example notebook for starters. Limited unit testing. Basic function documentation. Azure DevOps CI pipelines for `pylint` and `pytest` via the `mlos_core` `conda` `environment.yml`. Related work items: #274, #279
2022-03-14 22:37:52 +03:00
Makefile bdist rule tweaks (#316) Followup cleanup to #315
2023-04-26 02:46:16 +03:00
Introduce pyproject.toml build changes (#764) This PR builds off of #762 and #763 These are in part 1. followup fixups for #746 (e.g., to allow setuptools-scm to be pulled in at build time as a build dependency only when conda an pip have mismatched version issues), and 2. Modernization improvements to allow us to make better use of other tools (e.g., `black` that only accept `pyproject.toml` files as their configuration files). To do so, we move some configs from `setup.py` to `pyproject.toml` for each module. However, to retain the ability to rewrite URLs in published README.md files on PyPi as well as consistent version dependencies across modules without the need to manually specify version numbers (e.g., using `setuptools-scm`) we mark a few dependencies as dynamic and leave our existing logic inside the `setup.py` file. Finally, we reorganize the `version.py` file to be inside the module and fix a few previous omissions for `mlos_viz`.
2024-07-03 01:31:31 +03:00
# setuptools-scm needs a longer history than Github CI workers have by default.
.PHONY: unshallow
unshallow: build/unshallow.build-stamp
build/unshallow.build-stamp:
git rev-parse --is-shallow-repository | grep -x -q false || git fetch --unshallow --quiet
touch $@
Merged PR 333: Initial MlosCore Implementation Adds basic `register` and `suggest` APIs for a couple of optimizer backends: `Skopt`, `Emukit`, `Random`. Includes example notebook for starters. Limited unit testing. Basic function documentation. Azure DevOps CI pipelines for `pylint` and `pytest` via the `mlos_core` `conda` `environment.yml`. Related work items: #274, #279
2022-03-14 22:37:52 +03:00
.PHONY: dist
Introduce pyproject.toml build changes (#764) This PR builds off of #762 and #763 These are in part 1. followup fixups for #746 (e.g., to allow setuptools-scm to be pulled in at build time as a build dependency only when conda an pip have mismatched version issues), and 2. Modernization improvements to allow us to make better use of other tools (e.g., `black` that only accept `pyproject.toml` files as their configuration files). To do so, we move some configs from `setup.py` to `pyproject.toml` for each module. However, to retain the ability to rewrite URLs in published README.md files on PyPi as well as consistent version dependencies across modules without the need to manually specify version numbers (e.g., using `setuptools-scm`) we mark a few dependencies as dynamic and leave our existing logic inside the `setup.py` file. Finally, we reorganize the `version.py` file to be inside the module and fix a few previous omissions for `mlos_viz`.
2024-07-03 01:31:31 +03:00
dist: sdist bdist_wheel
.PHONY: sdist
sdist: conda-env unshallow
sdist: mlos_core/dist/tmp/mlos_core-latest.tar.gz
sdist: mlos_bench/dist/tmp/mlos_bench-latest.tar.gz
sdist: mlos_viz/dist/tmp/mlos_viz-latest.tar.gz
Merged PR 333: Initial MlosCore Implementation Adds basic `register` and `suggest` APIs for a couple of optimizer backends: `Skopt`, `Emukit`, `Random`. Includes example notebook for starters. Limited unit testing. Basic function documentation. Azure DevOps CI pipelines for `pylint` and `pytest` via the `mlos_core` `conda` `environment.yml`. Related work items: #274, #279
2022-03-14 22:37:52 +03:00
.PHONY: bdist_wheel
Introduce pyproject.toml build changes (#764) This PR builds off of #762 and #763 These are in part 1. followup fixups for #746 (e.g., to allow setuptools-scm to be pulled in at build time as a build dependency only when conda an pip have mismatched version issues), and 2. Modernization improvements to allow us to make better use of other tools (e.g., `black` that only accept `pyproject.toml` files as their configuration files). To do so, we move some configs from `setup.py` to `pyproject.toml` for each module. However, to retain the ability to rewrite URLs in published README.md files on PyPi as well as consistent version dependencies across modules without the need to manually specify version numbers (e.g., using `setuptools-scm`) we mark a few dependencies as dynamic and leave our existing logic inside the `setup.py` file. Finally, we reorganize the `version.py` file to be inside the module and fix a few previous omissions for `mlos_viz`.
2024-07-03 01:31:31 +03:00
bdist_wheel: conda-env unshallow
Introduce basic experiment visualization module mlos_viz via dabl (#624) Adds a basic `mlos_viz.plot(exp)` style API for simple visualizations of `ExperimentData` results relative to the experiment's objectives (building off of #628 and https://github.com/dabl/dabl/pull/335). Note: this PR currently omits unit tests for the new module due to the complexity of testing visualizations. We intend to add this in future PRs. There is however, a working example of its use here right now: https://github.com/Microsoft-CISL/sqlite-autotuning/pull/41 --------- Co-authored-by: Sergiy Matusevych <sergiy.matusevych@gmail.com>
2024-01-16 21:48:20 +03:00
bdist_wheel: mlos_core/dist/tmp/mlos_core-latest-py3-none-any.whl
bdist_wheel: mlos_bench/dist/tmp/mlos_bench-latest-py3-none-any.whl
bdist_wheel: mlos_viz/dist/tmp/mlos_viz-latest-py3-none-any.whl
Makefile bdist rule tweaks (#316) Followup cleanup to #315
2023-04-26 02:46:16 +03:00
Introduce pyproject.toml build changes (#764) This PR builds off of #762 and #763 These are in part 1. followup fixups for #746 (e.g., to allow setuptools-scm to be pulled in at build time as a build dependency only when conda an pip have mismatched version issues), and 2. Modernization improvements to allow us to make better use of other tools (e.g., `black` that only accept `pyproject.toml` files as their configuration files). To do so, we move some configs from `setup.py` to `pyproject.toml` for each module. However, to retain the ability to rewrite URLs in published README.md files on PyPi as well as consistent version dependencies across modules without the need to manually specify version numbers (e.g., using `setuptools-scm`) we mark a few dependencies as dynamic and leave our existing logic inside the `setup.py` file. Finally, we reorganize the `version.py` file to be inside the module and fix a few previous omissions for `mlos_viz`.
2024-07-03 01:31:31 +03:00
# Make the whl files depend on the .tar.gz files, mostly to prevent conflicts
# with shared use of the their build/ trees.
Makefile bdist rule tweaks (#316) Followup cleanup to #315
2023-04-26 02:46:16 +03:00
mlos_core/dist/tmp/mlos_core-latest-py3-none-any.whl: MODULE_NAME := mlos_core
CI fixups: changes in package naming via upstream libs (dash to underscore) (#733)
2024-05-11 01:31:37 +03:00
mlos_core/dist/tmp/mlos_core-latest-py3-none-any.whl: PACKAGE_NAME := mlos_core
Introduce pyproject.toml build changes (#764) This PR builds off of #762 and #763 These are in part 1. followup fixups for #746 (e.g., to allow setuptools-scm to be pulled in at build time as a build dependency only when conda an pip have mismatched version issues), and 2. Modernization improvements to allow us to make better use of other tools (e.g., `black` that only accept `pyproject.toml` files as their configuration files). To do so, we move some configs from `setup.py` to `pyproject.toml` for each module. However, to retain the ability to rewrite URLs in published README.md files on PyPi as well as consistent version dependencies across modules without the need to manually specify version numbers (e.g., using `setuptools-scm`) we mark a few dependencies as dynamic and leave our existing logic inside the `setup.py` file. Finally, we reorganize the `version.py` file to be inside the module and fix a few previous omissions for `mlos_viz`.
2024-07-03 01:31:31 +03:00
mlos_core/dist/tmp/mlos_core-latest-py3-none-any.whl: mlos_core/dist/tmp/mlos_core-latest.tar.gz
mlos_core/dist/tmp/mlos_core-latest.tar.gz: $(MLOS_CORE_CONF_FILES) $(MLOS_CORE_PYTHON_FILES)
mlos_core/dist/tmp/mlos_core-latest.tar.gz: MODULE_NAME := mlos_core
mlos_core/dist/tmp/mlos_core-latest.tar.gz: PACKAGE_NAME := mlos_core
Makefile bdist rule tweaks (#316) Followup cleanup to #315
2023-04-26 02:46:16 +03:00
mlos_bench/dist/tmp/mlos_bench-latest-py3-none-any.whl: MODULE_NAME := mlos_bench
CI fixups: changes in package naming via upstream libs (dash to underscore) (#733)
2024-05-11 01:31:37 +03:00
mlos_bench/dist/tmp/mlos_bench-latest-py3-none-any.whl: PACKAGE_NAME := mlos_bench
Introduce pyproject.toml build changes (#764) This PR builds off of #762 and #763 These are in part 1. followup fixups for #746 (e.g., to allow setuptools-scm to be pulled in at build time as a build dependency only when conda an pip have mismatched version issues), and 2. Modernization improvements to allow us to make better use of other tools (e.g., `black` that only accept `pyproject.toml` files as their configuration files). To do so, we move some configs from `setup.py` to `pyproject.toml` for each module. However, to retain the ability to rewrite URLs in published README.md files on PyPi as well as consistent version dependencies across modules without the need to manually specify version numbers (e.g., using `setuptools-scm`) we mark a few dependencies as dynamic and leave our existing logic inside the `setup.py` file. Finally, we reorganize the `version.py` file to be inside the module and fix a few previous omissions for `mlos_viz`.
2024-07-03 01:31:31 +03:00
mlos_bench/dist/tmp/mlos_bench-latest-py3-none-any.whl: mlos_bench/dist/tmp/mlos_bench-latest.tar.gz
mlos_bench/dist/tmp/mlos_bench-latest.tar.gz: $(MLOS_BENCH_CONF_FILES) $(MLOS_BENCH_PYTHON_FILES)
mlos_bench/dist/tmp/mlos_bench-latest.tar.gz: MODULE_NAME := mlos_bench
mlos_bench/dist/tmp/mlos_bench-latest.tar.gz: PACKAGE_NAME := mlos_bench
Makefile bdist rule tweaks (#316) Followup cleanup to #315
2023-04-26 02:46:16 +03:00
Introduce basic experiment visualization module mlos_viz via dabl (#624) Adds a basic `mlos_viz.plot(exp)` style API for simple visualizations of `ExperimentData` results relative to the experiment's objectives (building off of #628 and https://github.com/dabl/dabl/pull/335). Note: this PR currently omits unit tests for the new module due to the complexity of testing visualizations. We intend to add this in future PRs. There is however, a working example of its use here right now: https://github.com/Microsoft-CISL/sqlite-autotuning/pull/41 --------- Co-authored-by: Sergiy Matusevych <sergiy.matusevych@gmail.com>
2024-01-16 21:48:20 +03:00
mlos_viz/dist/tmp/mlos_viz-latest-py3-none-any.whl: MODULE_NAME := mlos_viz
CI fixups: changes in package naming via upstream libs (dash to underscore) (#733)
2024-05-11 01:31:37 +03:00
mlos_viz/dist/tmp/mlos_viz-latest-py3-none-any.whl: PACKAGE_NAME := mlos_viz
Introduce pyproject.toml build changes (#764) This PR builds off of #762 and #763 These are in part 1. followup fixups for #746 (e.g., to allow setuptools-scm to be pulled in at build time as a build dependency only when conda an pip have mismatched version issues), and 2. Modernization improvements to allow us to make better use of other tools (e.g., `black` that only accept `pyproject.toml` files as their configuration files). To do so, we move some configs from `setup.py` to `pyproject.toml` for each module. However, to retain the ability to rewrite URLs in published README.md files on PyPi as well as consistent version dependencies across modules without the need to manually specify version numbers (e.g., using `setuptools-scm`) we mark a few dependencies as dynamic and leave our existing logic inside the `setup.py` file. Finally, we reorganize the `version.py` file to be inside the module and fix a few previous omissions for `mlos_viz`.
2024-07-03 01:31:31 +03:00
mlos_viz/dist/tmp/mlos_viz-latest-py3-none-any.whl: mlos_viz/dist/tmp/mlos_viz-latest.tar.gz
mlos_viz/dist/tmp/mlos_viz-latest.tar.gz: $(MLOS_VIZ_CONF_FILES) $(MLOS_VIZ_PYTHON_FILES)
mlos_viz/dist/tmp/mlos_viz-latest.tar.gz: MODULE_NAME := mlos_viz
mlos_viz/dist/tmp/mlos_viz-latest.tar.gz: PACKAGE_NAME := mlos_viz
Introduce basic experiment visualization module mlos_viz via dabl (#624) Adds a basic `mlos_viz.plot(exp)` style API for simple visualizations of `ExperimentData` results relative to the experiment's objectives (building off of #628 and https://github.com/dabl/dabl/pull/335). Note: this PR currently omits unit tests for the new module due to the complexity of testing visualizations. We intend to add this in future PRs. There is however, a working example of its use here right now: https://github.com/Microsoft-CISL/sqlite-autotuning/pull/41 --------- Co-authored-by: Sergiy Matusevych <sergiy.matusevych@gmail.com>
2024-01-16 21:48:20 +03:00
Introduce pyproject.toml build changes (#764) This PR builds off of #762 and #763 These are in part 1. followup fixups for #746 (e.g., to allow setuptools-scm to be pulled in at build time as a build dependency only when conda an pip have mismatched version issues), and 2. Modernization improvements to allow us to make better use of other tools (e.g., `black` that only accept `pyproject.toml` files as their configuration files). To do so, we move some configs from `setup.py` to `pyproject.toml` for each module. However, to retain the ability to rewrite URLs in published README.md files on PyPi as well as consistent version dependencies across modules without the need to manually specify version numbers (e.g., using `setuptools-scm`) we mark a few dependencies as dynamic and leave our existing logic inside the `setup.py` file. Finally, we reorganize the `version.py` file to be inside the module and fix a few previous omissions for `mlos_viz`.
2024-07-03 01:31:31 +03:00
%-latest.tar.gz: build/conda-env.${CONDA_ENV_NAME}.build-stamp build/unshallow.build-stamp $(FORMAT_PREREQS)
mkdir -p $(MODULE_NAME)/dist/tmp
rm -f $(MODULE_NAME)/dist/$(PACKAGE_NAME)-*.tar{,.gz}
rm -f $(MODULE_NAME)/dist/tmp/$(PACKAGE_NAME)-latest.tar{,.gz}
rm -rf $(MODULE_NAME)/build/
rm -rf $(MODULE_NAME)/$(MODULE_NAME).egg-info/
cd $(MODULE_NAME)/ && conda run -n ${CONDA_ENV_NAME} python3 -m build --sdist
# Do some sanity checks on the sdist tarball output.
BASE_VERS=`conda run -n ${CONDA_ENV_NAME} python3 $(MODULE_NAME)/$(MODULE_NAME)/version.py | cut -d. -f-2 | egrep -x '[0-9.]+' || echo err-unknown-base-version` \
Bump version: 0.5.1 → 0.6.0 (#795) Especially for build system updates.
2024-07-15 20:46:05 +03:00
&& TAG_VERS=`git tag -l --sort=-version:refname | egrep -x '^v[0-9.]+' | head -n1 | sed 's/^v//' | cut -d. -f-2 | egrep -x '[0-9.]+' || echo err-unknown-tag-version` \
&& ls $(MODULE_NAME)/dist/$(PACKAGE_NAME)-*.tar.gz | grep -F -e $$BASE_VERS -e $$TAG_VERS
Introduce pyproject.toml build changes (#764) This PR builds off of #762 and #763 These are in part 1. followup fixups for #746 (e.g., to allow setuptools-scm to be pulled in at build time as a build dependency only when conda an pip have mismatched version issues), and 2. Modernization improvements to allow us to make better use of other tools (e.g., `black` that only accept `pyproject.toml` files as their configuration files). To do so, we move some configs from `setup.py` to `pyproject.toml` for each module. However, to retain the ability to rewrite URLs in published README.md files on PyPi as well as consistent version dependencies across modules without the need to manually specify version numbers (e.g., using `setuptools-scm`) we mark a few dependencies as dynamic and leave our existing logic inside the `setup.py` file. Finally, we reorganize the `version.py` file to be inside the module and fix a few previous omissions for `mlos_viz`.
2024-07-03 01:31:31 +03:00
# Make sure tests were excluded.
! ( tar tzf $(MODULE_NAME)/dist/$(PACKAGE_NAME)-*.tar.gz | grep -m1 tests/ )
# Make sure the py.typed marker file exists.
tar tzf $(MODULE_NAME)/dist/$(PACKAGE_NAME)-*.tar.gz | grep -m1 /py.typed
Storage schema evolution support (#906) # Pull Request ## Title Adds support for dynamically evolving the storage schema over time. --- ## Description - Adds support for dynamic schema evolution via [`alembic`](https://alembic.sqlalchemy.org) - Details in included README.md, but basically uses a local stub sqlite db to automatically generate migration scripts that can be invoked by `alembic`. - Adds a `--create-update-storage-schema-only` flag to `Launcher` to allow running this independently. - This may be useful for `mlos_benchd` cases where multiple instances trying to manipulate the schema at once could be problematic. With this one could run this part separately first and then launch parallel other processes separately. - ~Adds the trial_runner_id column to the trials table (for #720)~ (moved to a separate PR) - Closes #649 --- ## Type of Change - ✨ New feature --- ## Testing - Tests for this are somewhat hard to do in CI. I've tested this manually in this case. Details in the README.md --- --------- Co-authored-by: Sergiy Matusevych <sergiym@microsoft.com>
2025-01-04 06:34:30 +03:00
# Make sure the alembic scripts are included
[ "$(MODULE_NAME)" != "mlos_bench" ] || tar tzf $(MODULE_NAME)/dist/$(PACKAGE_NAME)-*.tar.gz | grep -m1 /storage/sql/alembic.ini
[ "$(MODULE_NAME)" != "mlos_bench" ] || tar tzf $(MODULE_NAME)/dist/$(PACKAGE_NAME)-*.tar.gz | grep -m1 /storage/sql/alembic/env.py
[ "$(MODULE_NAME)" != "mlos_bench" ] || tar tzf $(MODULE_NAME)/dist/$(PACKAGE_NAME)-*.tar.gz | grep -m1 /storage/sql/alembic/versions/.*py
Introduce pyproject.toml build changes (#764) This PR builds off of #762 and #763 These are in part 1. followup fixups for #746 (e.g., to allow setuptools-scm to be pulled in at build time as a build dependency only when conda an pip have mismatched version issues), and 2. Modernization improvements to allow us to make better use of other tools (e.g., `black` that only accept `pyproject.toml` files as their configuration files). To do so, we move some configs from `setup.py` to `pyproject.toml` for each module. However, to retain the ability to rewrite URLs in published README.md files on PyPi as well as consistent version dependencies across modules without the need to manually specify version numbers (e.g., using `setuptools-scm`) we mark a few dependencies as dynamic and leave our existing logic inside the `setup.py` file. Finally, we reorganize the `version.py` file to be inside the module and fix a few previous omissions for `mlos_viz`.
2024-07-03 01:31:31 +03:00
# Check to make sure the mlos_bench module has the config directory.
[ "$(MODULE_NAME)" != "mlos_bench" ] || tar tzf $(MODULE_NAME)/dist/$(PACKAGE_NAME)-*.tar.gz | grep -m1 mlos_bench/config/
cd $(MODULE_NAME)/dist/tmp && ln -s ../$(PACKAGE_NAME)-*.tar.gz $(PACKAGE_NAME)-latest.tar.gz
%-latest-py3-none-any.whl: build/conda-env.${CONDA_ENV_NAME}.build-stamp build/unshallow.build-stamp $(FORMAT_PREREQS)
Makefile bdist rule tweaks (#316) Followup cleanup to #315
2023-04-26 02:46:16 +03:00
mkdir -p $(MODULE_NAME)/dist/tmp
rm -f $(MODULE_NAME)/dist/$(MODULE_NAME)-*-py3-none-any.whl
rm -f $(MODULE_NAME)/dist/tmp/$(MODULE_NAME)-latest-py3-none-any.whl
Introduce pyproject.toml build changes (#764) This PR builds off of #762 and #763 These are in part 1. followup fixups for #746 (e.g., to allow setuptools-scm to be pulled in at build time as a build dependency only when conda an pip have mismatched version issues), and 2. Modernization improvements to allow us to make better use of other tools (e.g., `black` that only accept `pyproject.toml` files as their configuration files). To do so, we move some configs from `setup.py` to `pyproject.toml` for each module. However, to retain the ability to rewrite URLs in published README.md files on PyPi as well as consistent version dependencies across modules without the need to manually specify version numbers (e.g., using `setuptools-scm`) we mark a few dependencies as dynamic and leave our existing logic inside the `setup.py` file. Finally, we reorganize the `version.py` file to be inside the module and fix a few previous omissions for `mlos_viz`.
2024-07-03 01:31:31 +03:00
rm -rf $(MODULE_NAME)/build/
rm -rf $(MODULE_NAME)/$(MODULE_NAME).egg-info/
cd $(MODULE_NAME)/ && conda run -n ${CONDA_ENV_NAME} python3 -m build --wheel
# Do some sanity checks on the wheel output.
BASE_VERS=`conda run -n ${CONDA_ENV_NAME} python3 $(MODULE_NAME)/$(MODULE_NAME)/version.py | cut -d. -f-2 | egrep -o '^[0-9.]+' || echo err-unknown-base-version` \
Bump version: 0.5.1 → 0.6.0 (#795) Especially for build system updates.
2024-07-15 20:46:05 +03:00
&& TAG_VERS=`git tag -l --sort=-version:refname | egrep -x '^v[0-9.]+' | head -n1 | sed 's/^v//' | cut -d. -f-2 | egrep -x '[0-9.]+' || echo err-unknown-tag-version` \
&& ls $(MODULE_NAME)/dist/$(MODULE_NAME)-*-py3-none-any.whl | grep -F -e $$BASE_VERS -e $$TAG_VERS
Fixups to exclude tests from the wheels (#315) * fixups to exclude tests from the wheel file * also exclude the test files even from the sdist tars
2023-04-25 23:31:28 +03:00
# Check to make sure the tests were excluded from the wheel.
Makefile bdist rule tweaks (#316) Followup cleanup to #315
2023-04-26 02:46:16 +03:00
! ( unzip -t $(MODULE_NAME)/dist/$(MODULE_NAME)-*-py3-none-any.whl | grep -m1 tests/ )
Introduce pyproject.toml build changes (#764) This PR builds off of #762 and #763 These are in part 1. followup fixups for #746 (e.g., to allow setuptools-scm to be pulled in at build time as a build dependency only when conda an pip have mismatched version issues), and 2. Modernization improvements to allow us to make better use of other tools (e.g., `black` that only accept `pyproject.toml` files as their configuration files). To do so, we move some configs from `setup.py` to `pyproject.toml` for each module. However, to retain the ability to rewrite URLs in published README.md files on PyPi as well as consistent version dependencies across modules without the need to manually specify version numbers (e.g., using `setuptools-scm`) we mark a few dependencies as dynamic and leave our existing logic inside the `setup.py` file. Finally, we reorganize the `version.py` file to be inside the module and fix a few previous omissions for `mlos_viz`.
2024-07-03 01:31:31 +03:00
# Make sure the py.typed marker file exists.
unzip -t $(MODULE_NAME)/dist/$(MODULE_NAME)-*-py3-none-any.whl | grep -m1 /py.typed
Refactor config examples directory structure (#313) * reorg examples/configs to a more consistent layout and start documenting it * include the config examples in the package * distribute the config directory * fixups to exclude tests from the wheel file * also exclude the test files even from the sdist tars * test that the configs got included in the wheel * automatically add the modules included configs to the search path * fix inconsistent plurals * add conditional dependency for importlib_resources for python 3.8 * adding mock vm ops services * make the publicly accessible version a property * allow llamma tune to always reverse the default config * handle null categorical values and disallow duplicate values * add local mock exec service * more mock services and tunables fixups * pylint fixups, spelling, more tests for tunable groups loading, and make tunable groups indexable by tunable as well as by name * merge tunables upstream only * splitting out tweaks to covariant group names and null value processing for tunables * add a test for duplicate values being disallowed * adding some tests for changes to tunable assignment/definition behavior * mypy cleanup * rework covariant group merge compatibility check * return None not 'None' * fixup score calc
2023-05-04 01:45:50 +03:00
# Check to make sure the mlos_bench module has the config directory.
[ "$(MODULE_NAME)" != "mlos_bench" ] || unzip -t $(MODULE_NAME)/dist/$(MODULE_NAME)-*-py3-none-any.whl | grep -m1 mlos_bench/config/
Pypi packaging (#626) Co-authored-by: Sergiy Matusevych <sergiy.matusevych@gmail.com>
2024-01-10 01:40:15 +03:00
# Check to make sure the README contents made it into the package metadata.
Introduce pyproject.toml build changes (#764) This PR builds off of #762 and #763 These are in part 1. followup fixups for #746 (e.g., to allow setuptools-scm to be pulled in at build time as a build dependency only when conda an pip have mismatched version issues), and 2. Modernization improvements to allow us to make better use of other tools (e.g., `black` that only accept `pyproject.toml` files as their configuration files). To do so, we move some configs from `setup.py` to `pyproject.toml` for each module. However, to retain the ability to rewrite URLs in published README.md files on PyPi as well as consistent version dependencies across modules without the need to manually specify version numbers (e.g., using `setuptools-scm`) we mark a few dependencies as dynamic and leave our existing logic inside the `setup.py` file. Finally, we reorganize the `version.py` file to be inside the module and fix a few previous omissions for `mlos_viz`.
2024-07-03 01:31:31 +03:00
unzip -p $(MODULE_NAME)/dist/$(MODULE_NAME)-*-py3-none-any.whl */METADATA | egrep -v '^[A-Z][a-zA-Z-]+:' | grep -q -i '^# mlos'
# Also check that the they include the URL
unzip -p $(MODULE_NAME)/dist/$(MODULE_NAME)-*-py3-none-any.whl */METADATA | grep -q -e '](https://github.com/microsoft/MLOS/'
# Link it into place
cd $(MODULE_NAME)/dist/tmp && ln -s ../$(MODULE_NAME)-*-py3-none-any.whl $(MODULE_NAME)-latest-py3-none-any.whl
Makefile bdist rule tweaks (#316) Followup cleanup to #315
2023-04-26 02:46:16 +03:00
Staged Makefile Improvements (#762) - Introduce a new `make format` target that currently only calls `licenseheaders` but will call `isort` and `black` in the future - Introduce new variables to help make dependency tracking easier - Simplify `pytest` rules and allow running only a single module at a time. This PR is a precursor to pyproject.toml related build changes as well. --------- Co-authored-by: Sergiy Matusevych <sergiym@microsoft.com>
2024-07-01 18:57:17 +03:00
.PHONY: clean-dist-test-env
clean-dist-test-env:
Merged PR 600: Fixup wheel generation
2022-09-01 00:49:31 +03:00
# Remove any existing mlos-dist-test environment so we can start clean.
conda env remove -y ${CONDA_INFO_LEVEL} -n mlos-dist-test-$(PYTHON_VERSION) 2>/dev/null || true
Merged PR 903: Combination of additional build and doc fixups This PR was originally aimed at just fixing the intermittent documentation build errors. That issue ultimately stems from conflicts between conda and pip installs of two different versions of `mistune` dependency. Along the way I found a handful of other improvements for build, pipeline, and documentation generation things. - Make sure we do the nightly build regardless of whether code has changed - Don't build the docs in parallel by default (trying to avoid an intermittent build failure) - Add basic support for testing devcontainer manually outside for vscode - Stop doing parallel builds so it's easier to debug, and move doc build/publishing to the devcontainer - Split windows conda env off, stop using gpy from conda-forge, and just build it locally with clang, gcc, or msvc - Adjust the devcontainer build so that more dependencies are built upfront during the container build. This makes the process more cacheable so that we can just pull the upstream version and not have to build as much locally which could delay first launch. - Import part of a fix from elsewhere - Switch to docker-compose style devcontainer that automatically also starts an nginx container for viewing the generated docs - Update some docs and doc building - [X] pydocstyle fixups - [X] Add some log filtering of the `make doc` output for easier debugging (e.g. don't care about missing examples for now) - [X] doc link lint checks - DONE - [X] Split this out into multiple PRs - [X] SpaceAdapterFactory: !959 - [X] docstyle fixups: !960 - [ ] doc build fixups - [ ] devcontainer improvements (leaving the last two as this PR)
2023-02-27 08:42:22 +03:00
rm -f build/dist-test-env.$(PYTHON_VERSION).build-stamp
Merged PR 600: Fixup wheel generation
2022-09-01 00:49:31 +03:00
.PHONY: dist-test-env
Merged PR 903: Combination of additional build and doc fixups This PR was originally aimed at just fixing the intermittent documentation build errors. That issue ultimately stems from conflicts between conda and pip installs of two different versions of `mistune` dependency. Along the way I found a handful of other improvements for build, pipeline, and documentation generation things. - Make sure we do the nightly build regardless of whether code has changed - Don't build the docs in parallel by default (trying to avoid an intermittent build failure) - Add basic support for testing devcontainer manually outside for vscode - Stop doing parallel builds so it's easier to debug, and move doc build/publishing to the devcontainer - Split windows conda env off, stop using gpy from conda-forge, and just build it locally with clang, gcc, or msvc - Adjust the devcontainer build so that more dependencies are built upfront during the container build. This makes the process more cacheable so that we can just pull the upstream version and not have to build as much locally which could delay first launch. - Import part of a fix from elsewhere - Switch to docker-compose style devcontainer that automatically also starts an nginx container for viewing the generated docs - Update some docs and doc building - [X] pydocstyle fixups - [X] Add some log filtering of the `make doc` output for easier debugging (e.g. don't care about missing examples for now) - [X] doc link lint checks - DONE - [X] Split this out into multiple PRs - [X] SpaceAdapterFactory: !959 - [X] docstyle fixups: !960 - [ ] doc build fixups - [ ] devcontainer improvements (leaving the last two as this PR)
2023-02-27 08:42:22 +03:00
dist-test-env: dist build/dist-test-env.$(PYTHON_VERSION).build-stamp
Merged PR 600: Fixup wheel generation
2022-09-01 00:49:31 +03:00
Merged PR 903: Combination of additional build and doc fixups This PR was originally aimed at just fixing the intermittent documentation build errors. That issue ultimately stems from conflicts between conda and pip installs of two different versions of `mistune` dependency. Along the way I found a handful of other improvements for build, pipeline, and documentation generation things. - Make sure we do the nightly build regardless of whether code has changed - Don't build the docs in parallel by default (trying to avoid an intermittent build failure) - Add basic support for testing devcontainer manually outside for vscode - Stop doing parallel builds so it's easier to debug, and move doc build/publishing to the devcontainer - Split windows conda env off, stop using gpy from conda-forge, and just build it locally with clang, gcc, or msvc - Adjust the devcontainer build so that more dependencies are built upfront during the container build. This makes the process more cacheable so that we can just pull the upstream version and not have to build as much locally which could delay first launch. - Import part of a fix from elsewhere - Switch to docker-compose style devcontainer that automatically also starts an nginx container for viewing the generated docs - Update some docs and doc building - [X] pydocstyle fixups - [X] Add some log filtering of the `make doc` output for easier debugging (e.g. don't care about missing examples for now) - [X] doc link lint checks - DONE - [X] Split this out into multiple PRs - [X] SpaceAdapterFactory: !959 - [X] docstyle fixups: !960 - [ ] doc build fixups - [ ] devcontainer improvements (leaving the last two as this PR)
2023-02-27 08:42:22 +03:00
build/dist-test-env.$(PYTHON_VERSION).build-stamp: build/conda-env.${CONDA_ENV_NAME}.build-stamp
Merged PR 860: Various Build Fixups - add nightly builds so we can catch issues with upstream changes sooner - allow reading conda envs out of the environment - handle only setting one of CONDA_EXPERIMENTAL_SOLVER and CONDA_SOLVER envs - adjust conda envs to avoid python 3.11 for now - adjust Makefile to use the same version of python from the current CONDA_ENV for our test environment (this is quick way to respect that version constraint and only specify it in one place) - adjust the expected version of configspace - remove configspace from conda env file (upstream doesn't have the version we need outside of conda-forge) - move all other version restrictions to setup.py files - add a 'full' mode for installing all extra_requires for each package - use that in the conda envs instead Related to additional build/test issues discovered in !850
2023-01-31 02:49:26 +03:00
# Use the same version of python as the one we used to build the wheels.
MacOS test fixups and CI pipelines (#876) # Pull Request ## Title Fixes some test issues for Mac OS environments and enables CI pipelines. --- ## Description - [x] mypy fixups - [x] Enable MacOS build pipeline - [x] ~Enable MacOS devcontainer tests~ - [x] ~Needs `docker` enabled (which isn't currently possible)~ - [x] ~Related: add *basic* devcontainer build test and run for Windows as well (also isn't currently possible)~ - [x] Fixes MacOS tests - [x] Address parallel runner issues with output dir cleanup - [x] Fix ssh tests (also a parallelization issue) - [x] ~Publish multi arch docker images (amd64 and arm64) (can't see above)~ Closes #875 --- ## Type of Change - 🛠️ Bug fix - ✨ New feature - 🧪 Tests --- ## Testing Locally tested on MacOS, Windows, and Linux. --- ## Additional Notes (optional) Leaves some currently disabled stub code for creating devcontainers in the future on MacOS and Windows hosts. Unfortunately this isn't currently possible with the Github Action runners. Also did some cosmetic renaming of the CI pipelines for better descriptions. This affects the required tests to pass in the repo settings. Will adjust that after this is approved. ---
2024-11-05 20:24:50 +03:00
build/dist-test-env.$(PYTHON_VERSION).build-stamp: PYTHON_VERS_REQ=$(shell conda list -n ${CONDA_ENV_NAME} | egrep '^python\s+' | sed -r -e 's/^python[ \t]+//' | cut -d' ' -f1 | cut -d. -f1-2)
Introduce basic experiment visualization module mlos_viz via dabl (#624) Adds a basic `mlos_viz.plot(exp)` style API for simple visualizations of `ExperimentData` results relative to the experiment's objectives (building off of #628 and https://github.com/dabl/dabl/pull/335). Note: this PR currently omits unit tests for the new module due to the complexity of testing visualizations. We intend to add this in future PRs. There is however, a working example of its use here right now: https://github.com/Microsoft-CISL/sqlite-autotuning/pull/41 --------- Co-authored-by: Sergiy Matusevych <sergiy.matusevych@gmail.com>
2024-01-16 21:48:20 +03:00
build/dist-test-env.$(PYTHON_VERSION).build-stamp: mlos_core/dist/tmp/mlos_core-latest-py3-none-any.whl
build/dist-test-env.$(PYTHON_VERSION).build-stamp: mlos_bench/dist/tmp/mlos_bench-latest-py3-none-any.whl
build/dist-test-env.$(PYTHON_VERSION).build-stamp: mlos_viz/dist/tmp/mlos_viz-latest-py3-none-any.whl
Merged PR 600: Fixup wheel generation
2022-09-01 00:49:31 +03:00
# Create a clean test environment for checking the wheel files.
Staged Makefile Improvements (#762) - Introduce a new `make format` target that currently only calls `licenseheaders` but will call `isort` and `black` in the future - Introduce new variables to help make dependency tracking easier - Simplify `pytest` rules and allow running only a single module at a time. This PR is a precursor to pyproject.toml related build changes as well. --------- Co-authored-by: Sergiy Matusevych <sergiym@microsoft.com>
2024-07-01 18:57:17 +03:00
$(MAKE) clean-dist-test-env
Merged PR 860: Various Build Fixups - add nightly builds so we can catch issues with upstream changes sooner - allow reading conda envs out of the environment - handle only setting one of CONDA_EXPERIMENTAL_SOLVER and CONDA_SOLVER envs - adjust conda envs to avoid python 3.11 for now - adjust Makefile to use the same version of python from the current CONDA_ENV for our test environment (this is quick way to respect that version constraint and only specify it in one place) - adjust the expected version of configspace - remove configspace from conda env file (upstream doesn't have the version we need outside of conda-forge) - move all other version restrictions to setup.py files - add a 'full' mode for installing all extra_requires for each package - use that in the conda envs instead Related to additional build/test issues discovered in !850
2023-01-31 02:49:26 +03:00
conda create -y ${CONDA_INFO_LEVEL} -n mlos-dist-test-$(PYTHON_VERSION) python=$(PYTHON_VERS_REQ)
Add some additional conda dependencies to build wheels (#519) Fixes some uncached rebuild issues that are holding up some other PRs.
2023-09-29 01:25:02 +03:00
# Install some additional dependencies necessary for clean building some of the wheels.
conda install -y ${CONDA_INFO_LEVEL} -n mlos-dist-test-$(PYTHON_VERSION) swig libpq
Merged PR 600: Fixup wheel generation
2022-09-01 00:49:31 +03:00
# Test a clean install of the mlos_core wheel.
Temporarily avoid new version of FLAML (#840) See Also: #839 Also fixes `make dist-test` rules to workaround recently broken `pip` version interpretation when using symlinked whl files.
2024-08-14 21:56:12 +03:00
conda run -n mlos-dist-test-$(PYTHON_VERSION) pip install "`readlink -f mlos_core/dist/tmp/mlos_core-latest-py3-none-any.whl`[full-tests]"
Merged PR 600: Fixup wheel generation
2022-09-01 00:49:31 +03:00
# Test a clean install of the mlos_bench wheel.
Temporarily avoid new version of FLAML (#840) See Also: #839 Also fixes `make dist-test` rules to workaround recently broken `pip` version interpretation when using symlinked whl files.
2024-08-14 21:56:12 +03:00
conda run -n mlos-dist-test-$(PYTHON_VERSION) pip install "`readlink -f mlos_bench/dist/tmp/mlos_bench-latest-py3-none-any.whl`[full-tests]"
Refactor config examples directory structure (#313) * reorg examples/configs to a more consistent layout and start documenting it * include the config examples in the package * distribute the config directory * fixups to exclude tests from the wheel file * also exclude the test files even from the sdist tars * test that the configs got included in the wheel * automatically add the modules included configs to the search path * fix inconsistent plurals * add conditional dependency for importlib_resources for python 3.8 * adding mock vm ops services * make the publicly accessible version a property * allow llamma tune to always reverse the default config * handle null categorical values and disallow duplicate values * add local mock exec service * more mock services and tunables fixups * pylint fixups, spelling, more tests for tunable groups loading, and make tunable groups indexable by tunable as well as by name * merge tunables upstream only * splitting out tweaks to covariant group names and null value processing for tunables * add a test for duplicate values being disallowed * adding some tests for changes to tunable assignment/definition behavior * mypy cleanup * rework covariant group merge compatibility check * return None not 'None' * fixup score calc
2023-05-04 01:45:50 +03:00
# Test that the config dir for mlos_bench got distributed.
test -e `conda env list | grep "mlos-dist-test-$(PYTHON_VERSION) " | awk '{ print $$2 }'`/lib/python$(PYTHON_VERS_REQ)/site-packages/mlos_bench/config/README.md
Introduce basic experiment visualization module mlos_viz via dabl (#624) Adds a basic `mlos_viz.plot(exp)` style API for simple visualizations of `ExperimentData` results relative to the experiment's objectives (building off of #628 and https://github.com/dabl/dabl/pull/335). Note: this PR currently omits unit tests for the new module due to the complexity of testing visualizations. We intend to add this in future PRs. There is however, a working example of its use here right now: https://github.com/Microsoft-CISL/sqlite-autotuning/pull/41 --------- Co-authored-by: Sergiy Matusevych <sergiy.matusevych@gmail.com>
2024-01-16 21:48:20 +03:00
# Test a clean install of the mlos_viz wheel.
Temporarily avoid new version of FLAML (#840) See Also: #839 Also fixes `make dist-test` rules to workaround recently broken `pip` version interpretation when using symlinked whl files.
2024-08-14 21:56:12 +03:00
conda run -n mlos-dist-test-$(PYTHON_VERSION) pip install "`readlink -f mlos_viz/dist/tmp/mlos_viz-latest-py3-none-any.whl`[full-tests]"
Merged PR 1026: Split test and check rules and improve test coverage publishing in pipelines and docs - make test and check targets for mlos_core and mlos_bench independent for faster execution and more clear output for `make test` and `make check` iteration when only hacking on one module locally - ensures that the full set still runs in the pipeline though - add support for skipping code coverage locally by setting `SKIP_COVERAGE=true` to run tests faster - optionally copy the code coverage data into our static html site for publishing (plus nice badges) - also, rather than racing and publishing the test results and code coverage for all of the different sub jobs, we only run and upload coverage via `pytest-azurepipelines` for the main one we're interested in to make reporting easier - also includes which job the test came from in the results - some other general `Makefile` cleanup Future work: - [X] Split the single pipeline into two pipelines and make the Windows one optional (since it's usually the one that takes the longest and we care a little bit less about it now that we have decontainer support) !1040
2023-03-16 22:47:01 +03:00
touch $@
Merged PR 600: Fixup wheel generation
2022-09-01 00:49:31 +03:00
.PHONY: dist-test
Staged Makefile Improvements (#762) - Introduce a new `make format` target that currently only calls `licenseheaders` but will call `isort` and `black` in the future - Introduce new variables to help make dependency tracking easier - Simplify `pytest` rules and allow running only a single module at a time. This PR is a precursor to pyproject.toml related build changes as well. --------- Co-authored-by: Sergiy Matusevych <sergiym@microsoft.com>
2024-07-01 18:57:17 +03:00
#dist-test: clean-dist
Merged PR 903: Combination of additional build and doc fixups This PR was originally aimed at just fixing the intermittent documentation build errors. That issue ultimately stems from conflicts between conda and pip installs of two different versions of `mistune` dependency. Along the way I found a handful of other improvements for build, pipeline, and documentation generation things. - Make sure we do the nightly build regardless of whether code has changed - Don't build the docs in parallel by default (trying to avoid an intermittent build failure) - Add basic support for testing devcontainer manually outside for vscode - Stop doing parallel builds so it's easier to debug, and move doc build/publishing to the devcontainer - Split windows conda env off, stop using gpy from conda-forge, and just build it locally with clang, gcc, or msvc - Adjust the devcontainer build so that more dependencies are built upfront during the container build. This makes the process more cacheable so that we can just pull the upstream version and not have to build as much locally which could delay first launch. - Import part of a fix from elsewhere - Switch to docker-compose style devcontainer that automatically also starts an nginx container for viewing the generated docs - Update some docs and doc building - [X] pydocstyle fixups - [X] Add some log filtering of the `make doc` output for easier debugging (e.g. don't care about missing examples for now) - [X] doc link lint checks - DONE - [X] Split this out into multiple PRs - [X] SpaceAdapterFactory: !959 - [X] docstyle fixups: !960 - [ ] doc build fixups - [ ] devcontainer improvements (leaving the last two as this PR)
2023-02-27 08:42:22 +03:00
dist-test: dist-test-env build/dist-test.$(PYTHON_VERSION).build-stamp
Merged PR 600: Fixup wheel generation
2022-09-01 00:49:31 +03:00
# Unnecessary if we invoke it as "python3 -m pytest ..."
Merged PR 903: Combination of additional build and doc fixups This PR was originally aimed at just fixing the intermittent documentation build errors. That issue ultimately stems from conflicts between conda and pip installs of two different versions of `mistune` dependency. Along the way I found a handful of other improvements for build, pipeline, and documentation generation things. - Make sure we do the nightly build regardless of whether code has changed - Don't build the docs in parallel by default (trying to avoid an intermittent build failure) - Add basic support for testing devcontainer manually outside for vscode - Stop doing parallel builds so it's easier to debug, and move doc build/publishing to the devcontainer - Split windows conda env off, stop using gpy from conda-forge, and just build it locally with clang, gcc, or msvc - Adjust the devcontainer build so that more dependencies are built upfront during the container build. This makes the process more cacheable so that we can just pull the upstream version and not have to build as much locally which could delay first launch. - Import part of a fix from elsewhere - Switch to docker-compose style devcontainer that automatically also starts an nginx container for viewing the generated docs - Update some docs and doc building - [X] pydocstyle fixups - [X] Add some log filtering of the `make doc` output for easier debugging (e.g. don't care about missing examples for now) - [X] doc link lint checks - DONE - [X] Split this out into multiple PRs - [X] SpaceAdapterFactory: !959 - [X] docstyle fixups: !960 - [ ] doc build fixups - [ ] devcontainer improvements (leaving the last two as this PR)
2023-02-27 08:42:22 +03:00
build/dist-test.$(PYTHON_VERSION).build-stamp: $(PYTHON_FILES) build/dist-test-env.$(PYTHON_VERSION).build-stamp
Merged PR 600: Fixup wheel generation
2022-09-01 00:49:31 +03:00
# Make sure we're using the packages from the wheel.
# Note: this will pick up the local directory and change the output if we're using PYTHONPATH=.
conda run -n mlos-dist-test-$(PYTHON_VERSION) pip list --verbose | grep mlos-core | grep ' pip'
conda run -n mlos-dist-test-$(PYTHON_VERSION) pip list --verbose | grep mlos-bench | grep ' pip'
Introduce basic experiment visualization module mlos_viz via dabl (#624) Adds a basic `mlos_viz.plot(exp)` style API for simple visualizations of `ExperimentData` results relative to the experiment's objectives (building off of #628 and https://github.com/dabl/dabl/pull/335). Note: this PR currently omits unit tests for the new module due to the complexity of testing visualizations. We intend to add this in future PRs. There is however, a working example of its use here right now: https://github.com/Microsoft-CISL/sqlite-autotuning/pull/41 --------- Co-authored-by: Sergiy Matusevych <sergiy.matusevych@gmail.com>
2024-01-16 21:48:20 +03:00
conda run -n mlos-dist-test-$(PYTHON_VERSION) pip list --verbose | grep mlos-viz | grep ' pip'
Merged PR 600: Fixup wheel generation
2022-09-01 00:49:31 +03:00
# Run a simple test that uses the mlos_core wheel (full tests can be checked with `make test`).
Merged PR 939: 2. Unit tests for mlos_bench optimziation loop also, reorganized unit tests for mlos_bench into groups for optimziers, environments, and tunable parameters
2023-03-04 01:53:37 +03:00
conda run -n mlos-dist-test-$(PYTHON_VERSION) python3 -m pytest mlos_core/mlos_core/tests/spaces/spaces_test.py
Merged PR 600: Fixup wheel generation
2022-09-01 00:49:31 +03:00
# Run a simple test that uses the mlos_bench wheel (full tests can be checked with `make test`).
Refactor for consistent use of plural submodule names (#318) * Refactor for consistent use of plural submodule names (pulled out from work in #313)
2023-04-28 03:39:58 +03:00
conda run -n mlos-dist-test-$(PYTHON_VERSION) python3 -m pytest mlos_bench/mlos_bench/tests/environments/mock_env_test.py
Introduce pyproject.toml build changes (#764) This PR builds off of #762 and #763 These are in part 1. followup fixups for #746 (e.g., to allow setuptools-scm to be pulled in at build time as a build dependency only when conda an pip have mismatched version issues), and 2. Modernization improvements to allow us to make better use of other tools (e.g., `black` that only accept `pyproject.toml` files as their configuration files). To do so, we move some configs from `setup.py` to `pyproject.toml` for each module. However, to retain the ability to rewrite URLs in published README.md files on PyPi as well as consistent version dependencies across modules without the need to manually specify version numbers (e.g., using `setuptools-scm`) we mark a few dependencies as dynamic and leave our existing logic inside the `setup.py` file. Finally, we reorganize the `version.py` file to be inside the module and fix a few previous omissions for `mlos_viz`.
2024-07-03 01:31:31 +03:00
# Run a basic cli tool check.
conda run -n mlos-dist-test-$(PYTHON_VERSION) mlos_bench --help 2>&1 | grep '^usage: mlos_bench '
Introduce basic experiment visualization module mlos_viz via dabl (#624) Adds a basic `mlos_viz.plot(exp)` style API for simple visualizations of `ExperimentData` results relative to the experiment's objectives (building off of #628 and https://github.com/dabl/dabl/pull/335). Note: this PR currently omits unit tests for the new module due to the complexity of testing visualizations. We intend to add this in future PRs. There is however, a working example of its use here right now: https://github.com/Microsoft-CISL/sqlite-autotuning/pull/41 --------- Co-authored-by: Sergiy Matusevych <sergiy.matusevych@gmail.com>
2024-01-16 21:48:20 +03:00
# Run a simple test that uses the mlos_viz wheel (full tests can be checked with `make test`).
Improve mlos-viz for multiple repeats of a config and add tests (#633) - Mark `mlos_viz` as `typed` for `mypy` - Bump version - Mock calls to matplotlib/dabl for testing - Add plotting of top-N configs - Improve plots for handling repeat config trials via variance error bars --------- Co-authored-by: Sergiy Matusevych <sergiym@microsoft.com>
2024-01-29 04:46:44 +03:00
# To do that, we need the fixtures from mlos_bench, so make those available too.
PYTHONPATH=mlos_bench conda run -n mlos-dist-test-$(PYTHON_VERSION) python3 -m pytest mlos_viz/mlos_viz/tests/test_dabl_plot.py
Merged PR 1026: Split test and check rules and improve test coverage publishing in pipelines and docs - make test and check targets for mlos_core and mlos_bench independent for faster execution and more clear output for `make test` and `make check` iteration when only hacking on one module locally - ensures that the full set still runs in the pipeline though - add support for skipping code coverage locally by setting `SKIP_COVERAGE=true` to run tests faster - optionally copy the code coverage data into our static html site for publishing (plus nice badges) - also, rather than racing and publishing the test results and code coverage for all of the different sub jobs, we only run and upload coverage via `pytest-azurepipelines` for the main one we're interested in to make reporting easier - also includes which job the test came from in the results - some other general `Makefile` cleanup Future work: - [X] Split the single pipeline into two pipelines and make the Windows one optional (since it's usually the one that takes the longest and we care a little bit less about it now that we have decontainer support) !1040
2023-03-16 22:47:01 +03:00
touch $@
Merged PR 600: Fixup wheel generation
2022-09-01 00:49:31 +03:00
Staged Makefile Improvements (#762) - Introduce a new `make format` target that currently only calls `licenseheaders` but will call `isort` and `black` in the future - Introduce new variables to help make dependency tracking easier - Simplify `pytest` rules and allow running only a single module at a time. This PR is a precursor to pyproject.toml related build changes as well. --------- Co-authored-by: Sergiy Matusevych <sergiym@microsoft.com>
2024-07-01 18:57:17 +03:00
clean-dist-test: clean-dist-test-env
Merged PR 903: Combination of additional build and doc fixups This PR was originally aimed at just fixing the intermittent documentation build errors. That issue ultimately stems from conflicts between conda and pip installs of two different versions of `mistune` dependency. Along the way I found a handful of other improvements for build, pipeline, and documentation generation things. - Make sure we do the nightly build regardless of whether code has changed - Don't build the docs in parallel by default (trying to avoid an intermittent build failure) - Add basic support for testing devcontainer manually outside for vscode - Stop doing parallel builds so it's easier to debug, and move doc build/publishing to the devcontainer - Split windows conda env off, stop using gpy from conda-forge, and just build it locally with clang, gcc, or msvc - Adjust the devcontainer build so that more dependencies are built upfront during the container build. This makes the process more cacheable so that we can just pull the upstream version and not have to build as much locally which could delay first launch. - Import part of a fix from elsewhere - Switch to docker-compose style devcontainer that automatically also starts an nginx container for viewing the generated docs - Update some docs and doc building - [X] pydocstyle fixups - [X] Add some log filtering of the `make doc` output for easier debugging (e.g. don't care about missing examples for now) - [X] doc link lint checks - DONE - [X] Split this out into multiple PRs - [X] SpaceAdapterFactory: !959 - [X] docstyle fixups: !960 - [ ] doc build fixups - [ ] devcontainer improvements (leaving the last two as this PR)
2023-02-27 08:42:22 +03:00
rm -f build/dist-test-env.$(PYTHON_VERSION).build-stamp
Merged PR 333: Initial MlosCore Implementation Adds basic `register` and `suggest` APIs for a couple of optimizer backends: `Skopt`, `Emukit`, `Random`. Includes example notebook for starters. Limited unit testing. Basic function documentation. Azure DevOps CI pipelines for `pylint` and `pytest` via the `mlos_core` `conda` `environment.yml`. Related work items: #274, #279
2022-03-14 22:37:52 +03:00
Makefile bdist rule tweaks (#316) Followup cleanup to #315
2023-04-26 02:46:16 +03:00
Pypi packaging (#626) Co-authored-by: Sergiy Matusevych <sergiy.matusevych@gmail.com>
2024-01-10 01:40:15 +03:00
.PHONY: publish
publish: publish-pypi
.PHONY:
PyPi publish makefile rules fixup (#640) Prior Makefile rule consolidation tried to group dependencies for the pattern rule into separate lines for readability, but apparently this isn't allowed. This minor changes separates those out into a variable we can reference instead. Tested on a clean build tree locally.
2024-01-19 03:53:20 +03:00
publish-pypi-deps: build/publish-pypi-deps.${CONDA_ENV_NAME}.build-stamp
Pypi packaging (#626) Co-authored-by: Sergiy Matusevych <sergiy.matusevych@gmail.com>
2024-01-10 01:40:15 +03:00
build/publish-pypi-deps.${CONDA_ENV_NAME}.build-stamp: build/conda-env.${CONDA_ENV_NAME}.build-stamp
conda run -n ${CONDA_ENV_NAME} pip install -U twine
touch $@
PyPi publish makefile rules fixup (#640) Prior Makefile rule consolidation tried to group dependencies for the pattern rule into separate lines for readability, but apparently this isn't allowed. This minor changes separates those out into a variable we can reference instead. Tested on a clean build tree locally.
2024-01-19 03:53:20 +03:00
PUBLISH_DEPS := build/publish-pypi-deps.${CONDA_ENV_NAME}.build-stamp
PUBLISH_DEPS += build/pytest.${CONDA_ENV_NAME}.build-stamp
Introduce pyproject.toml build changes (#764) This PR builds off of #762 and #763 These are in part 1. followup fixups for #746 (e.g., to allow setuptools-scm to be pulled in at build time as a build dependency only when conda an pip have mismatched version issues), and 2. Modernization improvements to allow us to make better use of other tools (e.g., `black` that only accept `pyproject.toml` files as their configuration files). To do so, we move some configs from `setup.py` to `pyproject.toml` for each module. However, to retain the ability to rewrite URLs in published README.md files on PyPi as well as consistent version dependencies across modules without the need to manually specify version numbers (e.g., using `setuptools-scm`) we mark a few dependencies as dynamic and leave our existing logic inside the `setup.py` file. Finally, we reorganize the `version.py` file to be inside the module and fix a few previous omissions for `mlos_viz`.
2024-07-03 01:31:31 +03:00
PUBLISH_DEPS += mlos_core/dist/tmp/mlos_core-latest.tar.gz
PUBLISH_DEPS += mlos_bench/dist/tmp/mlos_bench-latest.tar.gz
PUBLISH_DEPS += mlos_viz/dist/tmp/mlos_viz-latest.tar.gz
Bump version: 0.5.1 → 0.6.0 (#795) Especially for build system updates.
2024-07-15 20:46:05 +03:00
PUBLISH_DEPS += mlos_core/dist/tmp/mlos_core-latest-py3-none-any.whl
PUBLISH_DEPS += mlos_bench/dist/tmp/mlos_bench-latest-py3-none-any.whl
PUBLISH_DEPS += mlos_viz/dist/tmp/mlos_viz-latest-py3-none-any.whl
PyPi publish makefile rules fixup (#640) Prior Makefile rule consolidation tried to group dependencies for the pattern rule into separate lines for readability, but apparently this isn't allowed. This minor changes separates those out into a variable we can reference instead. Tested on a clean build tree locally.
2024-01-19 03:53:20 +03:00
PUBLISH_DEPS += build/dist-test.$(PYTHON_VERSION).build-stamp
PUBLISH_DEPS += build/check-doc.build-stamp
PUBLISH_DEPS += build/linklint-doc.build-stamp
build/publish.${CONDA_ENV_NAME}.%.py.build-stamp: $(PUBLISH_DEPS)
Bump version: 0.5.1 → 0.6.0 (#795) Especially for build system updates.
2024-07-15 20:46:05 +03:00
# Basic sanity checks on files about to be published.
# Run "make clean-dist && make dist" if these fail.
# Check the tar count.
Introduce pyproject.toml build changes (#764) This PR builds off of #762 and #763 These are in part 1. followup fixups for #746 (e.g., to allow setuptools-scm to be pulled in at build time as a build dependency only when conda an pip have mismatched version issues), and 2. Modernization improvements to allow us to make better use of other tools (e.g., `black` that only accept `pyproject.toml` files as their configuration files). To do so, we move some configs from `setup.py` to `pyproject.toml` for each module. However, to retain the ability to rewrite URLs in published README.md files on PyPi as well as consistent version dependencies across modules without the need to manually specify version numbers (e.g., using `setuptools-scm`) we mark a few dependencies as dynamic and leave our existing logic inside the `setup.py` file. Finally, we reorganize the `version.py` file to be inside the module and fix a few previous omissions for `mlos_viz`.
2024-07-03 01:31:31 +03:00
test `ls -1 mlos_core/dist/*.tar.gz | wc -l` -eq 1
test `ls -1 mlos_bench/dist/*.tar.gz | wc -l` -eq 1
test `ls -1 mlos_viz/dist/*.tar.gz | wc -l` -eq 1
test `ls -1 mlos_*/dist/*.tar.gz | wc -l` -eq 3
Bump version: 0.5.1 → 0.6.0 (#795) Especially for build system updates.
2024-07-15 20:46:05 +03:00
# Check the whl count.
test `ls -1 mlos_core/dist/*.whl | wc -l` -eq 1
test `ls -1 mlos_bench/dist/*.whl | wc -l` -eq 1
test `ls -1 mlos_viz/dist/*.whl | wc -l` -eq 1
test `ls -1 mlos_*/dist/*.whl | wc -l` -eq 3
# Publish the files to the specified repository.
PyPi publish makefile rules fixup (#640) Prior Makefile rule consolidation tried to group dependencies for the pattern rule into separate lines for readability, but apparently this isn't allowed. This minor changes separates those out into a variable we can reference instead. Tested on a clean build tree locally.
2024-01-19 03:53:20 +03:00
repo_name=`echo "$@" | sed -r -e 's|build/publish\.[^.]+\.||' -e 's|\.py\.build-stamp||'` \
Pypi packaging (#626) Co-authored-by: Sergiy Matusevych <sergiy.matusevych@gmail.com>
2024-01-10 01:40:15 +03:00
&& conda run -n ${CONDA_ENV_NAME} python3 -m twine upload --repository $$repo_name \
mlos_*/dist/mlos*-*.tar.gz mlos_*/dist/mlos*-*.whl
touch $@
PyPi publish makefile rules fixup (#640) Prior Makefile rule consolidation tried to group dependencies for the pattern rule into separate lines for readability, but apparently this isn't allowed. This minor changes separates those out into a variable we can reference instead. Tested on a clean build tree locally.
2024-01-19 03:53:20 +03:00
publish-pypi: build/publish.${CONDA_ENV_NAME}.pypi.py.build-stamp
publish-test-pypi: build/publish.${CONDA_ENV_NAME}.testpypi.py.build-stamp
Pypi packaging (#626) Co-authored-by: Sergiy Matusevych <sergiy.matusevych@gmail.com>
2024-01-10 01:40:15 +03:00
Fixups to example notebook for new APIs (#834) - [x] enable notebook execution in CI pipelines to prevent these mistakes in the future --------- Co-authored-by: Sergiy Matusevych <sergiym@microsoft.com>
2024-08-08 02:20:17 +03:00
notebook-exec-test: build/notebook-exec-test.${CONDA_ENV_NAME}.build-stamp
build/notebook-exec-test.${CONDA_ENV_NAME}.build-stamp: build/conda-env.${CONDA_ENV_NAME}.build-stamp $(NOTEBOOK_FILES)
find . -name '*.ipynb' -not -path '*/build/*' -print0 | xargs -0 -n1 -P0 conda run -n ${CONDA_ENV_NAME} python -m jupyter execute
touch $@
Staged Makefile Improvements (#762) - Introduce a new `make format` target that currently only calls `licenseheaders` but will call `isort` and `black` in the future - Introduce new variables to help make dependency tracking easier - Simplify `pytest` rules and allow running only a single module at a time. This PR is a precursor to pyproject.toml related build changes as well. --------- Co-authored-by: Sergiy Matusevych <sergiym@microsoft.com>
2024-07-01 18:57:17 +03:00
Merged PR 903: Combination of additional build and doc fixups This PR was originally aimed at just fixing the intermittent documentation build errors. That issue ultimately stems from conflicts between conda and pip installs of two different versions of `mistune` dependency. Along the way I found a handful of other improvements for build, pipeline, and documentation generation things. - Make sure we do the nightly build regardless of whether code has changed - Don't build the docs in parallel by default (trying to avoid an intermittent build failure) - Add basic support for testing devcontainer manually outside for vscode - Stop doing parallel builds so it's easier to debug, and move doc build/publishing to the devcontainer - Split windows conda env off, stop using gpy from conda-forge, and just build it locally with clang, gcc, or msvc - Adjust the devcontainer build so that more dependencies are built upfront during the container build. This makes the process more cacheable so that we can just pull the upstream version and not have to build as much locally which could delay first launch. - Import part of a fix from elsewhere - Switch to docker-compose style devcontainer that automatically also starts an nginx container for viewing the generated docs - Update some docs and doc building - [X] pydocstyle fixups - [X] Add some log filtering of the `make doc` output for easier debugging (e.g. don't care about missing examples for now) - [X] doc link lint checks - DONE - [X] Split this out into multiple PRs - [X] SpaceAdapterFactory: !959 - [X] docstyle fixups: !960 - [ ] doc build fixups - [ ] devcontainer improvements (leaving the last two as this PR)
2023-02-27 08:42:22 +03:00
build/doc-prereqs.${CONDA_ENV_NAME}.build-stamp: build/conda-env.${CONDA_ENV_NAME}.build-stamp
build/doc-prereqs.${CONDA_ENV_NAME}.build-stamp: doc/requirements.txt
Merged PR 860: Various Build Fixups - add nightly builds so we can catch issues with upstream changes sooner - allow reading conda envs out of the environment - handle only setting one of CONDA_EXPERIMENTAL_SOLVER and CONDA_SOLVER envs - adjust conda envs to avoid python 3.11 for now - adjust Makefile to use the same version of python from the current CONDA_ENV for our test environment (this is quick way to respect that version constraint and only specify it in one place) - adjust the expected version of configspace - remove configspace from conda env file (upstream doesn't have the version we need outside of conda-forge) - move all other version restrictions to setup.py files - add a 'full' mode for installing all extra_requires for each package - use that in the conda envs instead Related to additional build/test issues discovered in !850
2023-01-31 02:49:26 +03:00
conda run -n ${CONDA_ENV_NAME} pip install -U -r doc/requirements.txt
Merged PR 1026: Split test and check rules and improve test coverage publishing in pipelines and docs - make test and check targets for mlos_core and mlos_bench independent for faster execution and more clear output for `make test` and `make check` iteration when only hacking on one module locally - ensures that the full set still runs in the pipeline though - add support for skipping code coverage locally by setting `SKIP_COVERAGE=true` to run tests faster - optionally copy the code coverage data into our static html site for publishing (plus nice badges) - also, rather than racing and publishing the test results and code coverage for all of the different sub jobs, we only run and upload coverage via `pytest-azurepipelines` for the main one we're interested in to make reporting easier - also includes which job the test came from in the results - some other general `Makefile` cleanup Future work: - [X] Split the single pipeline into two pipelines and make the Windows one optional (since it's usually the one that takes the longest and we care a little bit less about it now that we have decontainer support) !1040
2023-03-16 22:47:01 +03:00
touch $@
Merged PR 386: Generate and publish documentation from python docstrings - initial attempts at generating documentation Related work items: #307
2022-03-17 04:33:28 +03:00
.PHONY: doc-prereqs
Introduce pyproject.toml build changes (#764) This PR builds off of #762 and #763 These are in part 1. followup fixups for #746 (e.g., to allow setuptools-scm to be pulled in at build time as a build dependency only when conda an pip have mismatched version issues), and 2. Modernization improvements to allow us to make better use of other tools (e.g., `black` that only accept `pyproject.toml` files as their configuration files). To do so, we move some configs from `setup.py` to `pyproject.toml` for each module. However, to retain the ability to rewrite URLs in published README.md files on PyPi as well as consistent version dependencies across modules without the need to manually specify version numbers (e.g., using `setuptools-scm`) we mark a few dependencies as dynamic and leave our existing logic inside the `setup.py` file. Finally, we reorganize the `version.py` file to be inside the module and fix a few previous omissions for `mlos_viz`.
2024-07-03 01:31:31 +03:00
doc-prereqs: build/doc-prereqs.${CONDA_ENV_NAME}.build-stamp build/unshallow.build-stamp
Merged PR 386: Generate and publish documentation from python docstrings - initial attempts at generating documentation Related work items: #307
2022-03-17 04:33:28 +03:00
Merged PR 561: Reorganize file layout to support separate whl outputs Related work items: #444
2022-08-04 23:26:40 +03:00
.PHONY: clean-doc-env
clean-doc-env:
Merged PR 903: Combination of additional build and doc fixups This PR was originally aimed at just fixing the intermittent documentation build errors. That issue ultimately stems from conflicts between conda and pip installs of two different versions of `mistune` dependency. Along the way I found a handful of other improvements for build, pipeline, and documentation generation things. - Make sure we do the nightly build regardless of whether code has changed - Don't build the docs in parallel by default (trying to avoid an intermittent build failure) - Add basic support for testing devcontainer manually outside for vscode - Stop doing parallel builds so it's easier to debug, and move doc build/publishing to the devcontainer - Split windows conda env off, stop using gpy from conda-forge, and just build it locally with clang, gcc, or msvc - Adjust the devcontainer build so that more dependencies are built upfront during the container build. This makes the process more cacheable so that we can just pull the upstream version and not have to build as much locally which could delay first launch. - Import part of a fix from elsewhere - Switch to docker-compose style devcontainer that automatically also starts an nginx container for viewing the generated docs - Update some docs and doc building - [X] pydocstyle fixups - [X] Add some log filtering of the `make doc` output for easier debugging (e.g. don't care about missing examples for now) - [X] doc link lint checks - DONE - [X] Split this out into multiple PRs - [X] SpaceAdapterFactory: !959 - [X] docstyle fixups: !960 - [ ] doc build fixups - [ ] devcontainer improvements (leaving the last two as this PR)
2023-02-27 08:42:22 +03:00
rm -f build/doc-prereqs.build-stamp
rm -f build/doc-prereqs.${CONDA_ENV_NAME}.build-stamp
Merged PR 561: Reorganize file layout to support separate whl outputs Related work items: #444
2022-08-04 23:26:40 +03:00
Merged PR 903: Combination of additional build and doc fixups This PR was originally aimed at just fixing the intermittent documentation build errors. That issue ultimately stems from conflicts between conda and pip installs of two different versions of `mistune` dependency. Along the way I found a handful of other improvements for build, pipeline, and documentation generation things. - Make sure we do the nightly build regardless of whether code has changed - Don't build the docs in parallel by default (trying to avoid an intermittent build failure) - Add basic support for testing devcontainer manually outside for vscode - Stop doing parallel builds so it's easier to debug, and move doc build/publishing to the devcontainer - Split windows conda env off, stop using gpy from conda-forge, and just build it locally with clang, gcc, or msvc - Adjust the devcontainer build so that more dependencies are built upfront during the container build. This makes the process more cacheable so that we can just pull the upstream version and not have to build as much locally which could delay first launch. - Import part of a fix from elsewhere - Switch to docker-compose style devcontainer that automatically also starts an nginx container for viewing the generated docs - Update some docs and doc building - [X] pydocstyle fixups - [X] Add some log filtering of the `make doc` output for easier debugging (e.g. don't care about missing examples for now) - [X] doc link lint checks - DONE - [X] Split this out into multiple PRs - [X] SpaceAdapterFactory: !959 - [X] docstyle fixups: !960 - [ ] doc build fixups - [ ] devcontainer improvements (leaving the last two as this PR)
2023-02-27 08:42:22 +03:00
COMMON_DOC_FILES := build/doc-prereqs.${CONDA_ENV_NAME}.build-stamp doc/source/*.rst doc/source/_templates/*.rst doc/source/conf.py
Rework documentation generation (#869) # Pull Request ## Title Documentation generation rework. --- ## Description This PR does an initial revamp of the way we generate and write documentation. The goal is to simplify the generation process a bit by removing some hacks but also enable more strict checking of the documentation produced so that links resolve properly so that it is easier to navigate. This PR is mostly the mechanics and the associated fixups to do that. Future PRs can handle additional documentation in docstrings. Below is a more detailed description of some of the changes. - [x] Apply `black` and `pylint` to the sphinx `conf.py` - [x] Treat sphinx warnings as errors - [x] Reorg doc generation using `autoapi` to handle cross referencing warnings and drop use of `sphinx-apidoc` - [x] ~Update `overview.rst` to follow suite~ - [x] Alternatively, move overview text into the docstrings and "just" use `autoapi` for everything. - [x] Use `:py:func:`, `:py:class:`, `:py:meth:`, `:py:mod:`, etc. to cross reference functions and classes in the docstrings. - [x] Add `napolean` to parse `numpydoc` style docstrings (previous `numpydoc` method was incomplete) - [x] Use `intersphinx` and `nitpick` to check (or ignore where not possible) all internal and most external cross references to ensure good code navigation - [x] Fixup all broken references by either adding fully qualified types in docstrings or in the imports. - [x] Ignore type aliase for for now (documented bug, will address in a future PR) - [ ] Add a few high level examples in docstrings. - [ ] May need `from_config_string` style loaders to help with this. - [x] Add `doctest` to `pytest` to validate examples. - [x] Update `mlos_bench.run` help usage output to an `rst` file and link that into the man table of contents tree in the output. --- ## Type of Change - 🛠️ Bug fix - 📝 Documentation update - 🧪 Tests --- ## Testing - Adds additional checks and linting. - Must pass CI - Manual testing as follows: ```sh # SKIP_COVERAGE tweak is optional - just avoids a `pytest` job dependency make SKIP_COVERAGE=true doc ./doc/nginx-docker.sh restart ``` Browse to `http://localhost:8080` and check the results. --- ## Additional Notes Future PRs can add additional documentation strings for the `mlos_bench` classes including examples. In particular - [ ] mlos_bench.optimizers - [ ] mlos_bench.services --------- Co-authored-by: Sergiy Matusevych <sergiy.matusevych@gmail.com>
2024-11-15 21:59:31 +03:00
SPHINX_API_RST_FILES := doc/source/index.rst doc/source/mlos_bench.run.usage.rst
Merged PR 903: Combination of additional build and doc fixups This PR was originally aimed at just fixing the intermittent documentation build errors. That issue ultimately stems from conflicts between conda and pip installs of two different versions of `mistune` dependency. Along the way I found a handful of other improvements for build, pipeline, and documentation generation things. - Make sure we do the nightly build regardless of whether code has changed - Don't build the docs in parallel by default (trying to avoid an intermittent build failure) - Add basic support for testing devcontainer manually outside for vscode - Stop doing parallel builds so it's easier to debug, and move doc build/publishing to the devcontainer - Split windows conda env off, stop using gpy from conda-forge, and just build it locally with clang, gcc, or msvc - Adjust the devcontainer build so that more dependencies are built upfront during the container build. This makes the process more cacheable so that we can just pull the upstream version and not have to build as much locally which could delay first launch. - Import part of a fix from elsewhere - Switch to docker-compose style devcontainer that automatically also starts an nginx container for viewing the generated docs - Update some docs and doc building - [X] pydocstyle fixups - [X] Add some log filtering of the `make doc` output for easier debugging (e.g. don't care about missing examples for now) - [X] doc link lint checks - DONE - [X] Split this out into multiple PRs - [X] SpaceAdapterFactory: !959 - [X] docstyle fixups: !960 - [ ] doc build fixups - [ ] devcontainer improvements (leaving the last two as this PR)
2023-02-27 08:42:22 +03:00
Merged PR 1026: Split test and check rules and improve test coverage publishing in pipelines and docs - make test and check targets for mlos_core and mlos_bench independent for faster execution and more clear output for `make test` and `make check` iteration when only hacking on one module locally - ensures that the full set still runs in the pipeline though - add support for skipping code coverage locally by setting `SKIP_COVERAGE=true` to run tests faster - optionally copy the code coverage data into our static html site for publishing (plus nice badges) - also, rather than racing and publishing the test results and code coverage for all of the different sub jobs, we only run and upload coverage via `pytest-azurepipelines` for the main one we're interested in to make reporting easier - also includes which job the test came from in the results - some other general `Makefile` cleanup Future work: - [X] Split the single pipeline into two pipelines and make the Windows one optional (since it's usually the one that takes the longest and we care a little bit less about it now that we have decontainer support) !1040
2023-03-16 22:47:01 +03:00
ifeq ($(SKIP_COVERAGE),)
doc/build/html/index.html: build/pytest.${CONDA_ENV_NAME}.build-stamp
doc/build/html/htmlcov/index.html: build/pytest.${CONDA_ENV_NAME}.build-stamp
endif
Rework documentation generation (#869) # Pull Request ## Title Documentation generation rework. --- ## Description This PR does an initial revamp of the way we generate and write documentation. The goal is to simplify the generation process a bit by removing some hacks but also enable more strict checking of the documentation produced so that links resolve properly so that it is easier to navigate. This PR is mostly the mechanics and the associated fixups to do that. Future PRs can handle additional documentation in docstrings. Below is a more detailed description of some of the changes. - [x] Apply `black` and `pylint` to the sphinx `conf.py` - [x] Treat sphinx warnings as errors - [x] Reorg doc generation using `autoapi` to handle cross referencing warnings and drop use of `sphinx-apidoc` - [x] ~Update `overview.rst` to follow suite~ - [x] Alternatively, move overview text into the docstrings and "just" use `autoapi` for everything. - [x] Use `:py:func:`, `:py:class:`, `:py:meth:`, `:py:mod:`, etc. to cross reference functions and classes in the docstrings. - [x] Add `napolean` to parse `numpydoc` style docstrings (previous `numpydoc` method was incomplete) - [x] Use `intersphinx` and `nitpick` to check (or ignore where not possible) all internal and most external cross references to ensure good code navigation - [x] Fixup all broken references by either adding fully qualified types in docstrings or in the imports. - [x] Ignore type aliase for for now (documented bug, will address in a future PR) - [ ] Add a few high level examples in docstrings. - [ ] May need `from_config_string` style loaders to help with this. - [x] Add `doctest` to `pytest` to validate examples. - [x] Update `mlos_bench.run` help usage output to an `rst` file and link that into the man table of contents tree in the output. --- ## Type of Change - 🛠️ Bug fix - 📝 Documentation update - 🧪 Tests --- ## Testing - Adds additional checks and linting. - Must pass CI - Manual testing as follows: ```sh # SKIP_COVERAGE tweak is optional - just avoids a `pytest` job dependency make SKIP_COVERAGE=true doc ./doc/nginx-docker.sh restart ``` Browse to `http://localhost:8080` and check the results. --- ## Additional Notes Future PRs can add additional documentation strings for the `mlos_bench` classes including examples. In particular - [ ] mlos_bench.optimizers - [ ] mlos_bench.services --------- Co-authored-by: Sergiy Matusevych <sergiy.matusevych@gmail.com>
2024-11-15 21:59:31 +03:00
# Treat warnings as failures.
SPHINXOPTS ?= # -v # be verbose
SPHINXOPTS += -n -W -w $(CURDIR)/doc/build/sphinx-build.warn.log -j auto
doc/source/generated/mlos_bench.run.usage.txt: build/conda-env.${CONDA_ENV_NAME}.build-stamp
doc/source/generated/mlos_bench.run.usage.txt: $(MLOS_BENCH_PYTHON_FILES)
# Generate the help output from mlos_bench CLI for the docs.
mkdir -p doc/source/generated/
conda run -n ${CONDA_ENV_NAME} mlos_bench --help > doc/source/generated/mlos_bench.run.usage.txt
doc/build/html/index.html: build/doc-prereqs.${CONDA_ENV_NAME}.build-stamp
doc/build/html/index.html: doc/source/generated/mlos_bench.run.usage.txt
doc/build/html/index.html: $(MLOS_CORE_PYTHON_FILES)
doc/build/html/index.html: $(MLOS_BENCH_PYTHON_FILES)
doc/build/html/index.html: $(MLOS_VIZ_PYTHON_FILES)
doc/build/html/index.html: $(SPHINX_API_RST_FILES) doc/Makefile doc/source/conf.py
doc/build/html/index.html: doc/copy-source-tree-docs.sh $(MD_FILES)
More documentation updates (#887) # Pull Request ## Title More documentation updates --- ## Description - [x] Adds general documentation on philosophy and usage - [x] Additional docs about Config structure - [x] Adjusts README documentation links - [x] Additional documentation about Storage classes --- ## Type of Change - 📝 Documentation update - 🧪 Tests --- ## Additional Notes (optional) Follow on to #869 ---
2024-12-06 00:34:52 +03:00
#@rm -rf doc/build # let us cache things
Merged PR 903: Combination of additional build and doc fixups This PR was originally aimed at just fixing the intermittent documentation build errors. That issue ultimately stems from conflicts between conda and pip installs of two different versions of `mistune` dependency. Along the way I found a handful of other improvements for build, pipeline, and documentation generation things. - Make sure we do the nightly build regardless of whether code has changed - Don't build the docs in parallel by default (trying to avoid an intermittent build failure) - Add basic support for testing devcontainer manually outside for vscode - Stop doing parallel builds so it's easier to debug, and move doc build/publishing to the devcontainer - Split windows conda env off, stop using gpy from conda-forge, and just build it locally with clang, gcc, or msvc - Adjust the devcontainer build so that more dependencies are built upfront during the container build. This makes the process more cacheable so that we can just pull the upstream version and not have to build as much locally which could delay first launch. - Import part of a fix from elsewhere - Switch to docker-compose style devcontainer that automatically also starts an nginx container for viewing the generated docs - Update some docs and doc building - [X] pydocstyle fixups - [X] Add some log filtering of the `make doc` output for easier debugging (e.g. don't care about missing examples for now) - [X] doc link lint checks - DONE - [X] Split this out into multiple PRs - [X] SpaceAdapterFactory: !959 - [X] docstyle fixups: !960 - [ ] doc build fixups - [ ] devcontainer improvements (leaving the last two as this PR)
2023-02-27 08:42:22 +03:00
@mkdir -p doc/build
@rm -f doc/build/log.txt
Merged PR 1026: Split test and check rules and improve test coverage publishing in pipelines and docs - make test and check targets for mlos_core and mlos_bench independent for faster execution and more clear output for `make test` and `make check` iteration when only hacking on one module locally - ensures that the full set still runs in the pipeline though - add support for skipping code coverage locally by setting `SKIP_COVERAGE=true` to run tests faster - optionally copy the code coverage data into our static html site for publishing (plus nice badges) - also, rather than racing and publishing the test results and code coverage for all of the different sub jobs, we only run and upload coverage via `pytest-azurepipelines` for the main one we're interested in to make reporting easier - also includes which job the test came from in the results - some other general `Makefile` cleanup Future work: - [X] Split the single pipeline into two pipelines and make the Windows one optional (since it's usually the one that takes the longest and we care a little bit less about it now that we have decontainer support) !1040
2023-03-16 22:47:01 +03:00
# Make sure that at least placeholder doc badges are there.
mkdir -p doc/source/badges/
touch doc/source/badges/coverage.svg
touch doc/source/badges/tests.svg
experiment_id documentation improvements (#601) Closes #599
2023-11-20 23:50:49 +03:00
# Copy some of the source tree markdown docs into place.
Some documentation fixups (#524) - fix a quoting issue in an example - cross link the `sqlite-autotuning` repo - reuse the content from the main README.md's for the github pages site - general improvements to the landing page docs --------- Co-authored-by: Sergiy Matusevych <sergiym@microsoft.com>
2023-11-08 03:11:57 +03:00
./doc/copy-source-tree-docs.sh
Merged PR 903: Combination of additional build and doc fixups This PR was originally aimed at just fixing the intermittent documentation build errors. That issue ultimately stems from conflicts between conda and pip installs of two different versions of `mistune` dependency. Along the way I found a handful of other improvements for build, pipeline, and documentation generation things. - Make sure we do the nightly build regardless of whether code has changed - Don't build the docs in parallel by default (trying to avoid an intermittent build failure) - Add basic support for testing devcontainer manually outside for vscode - Stop doing parallel builds so it's easier to debug, and move doc build/publishing to the devcontainer - Split windows conda env off, stop using gpy from conda-forge, and just build it locally with clang, gcc, or msvc - Adjust the devcontainer build so that more dependencies are built upfront during the container build. This makes the process more cacheable so that we can just pull the upstream version and not have to build as much locally which could delay first launch. - Import part of a fix from elsewhere - Switch to docker-compose style devcontainer that automatically also starts an nginx container for viewing the generated docs - Update some docs and doc building - [X] pydocstyle fixups - [X] Add some log filtering of the `make doc` output for easier debugging (e.g. don't care about missing examples for now) - [X] doc link lint checks - DONE - [X] Split this out into multiple PRs - [X] SpaceAdapterFactory: !959 - [X] docstyle fixups: !960 - [ ] doc build fixups - [ ] devcontainer improvements (leaving the last two as this PR)
2023-02-27 08:42:22 +03:00
# Build the rst files into html.
Rework documentation generation (#869) # Pull Request ## Title Documentation generation rework. --- ## Description This PR does an initial revamp of the way we generate and write documentation. The goal is to simplify the generation process a bit by removing some hacks but also enable more strict checking of the documentation produced so that links resolve properly so that it is easier to navigate. This PR is mostly the mechanics and the associated fixups to do that. Future PRs can handle additional documentation in docstrings. Below is a more detailed description of some of the changes. - [x] Apply `black` and `pylint` to the sphinx `conf.py` - [x] Treat sphinx warnings as errors - [x] Reorg doc generation using `autoapi` to handle cross referencing warnings and drop use of `sphinx-apidoc` - [x] ~Update `overview.rst` to follow suite~ - [x] Alternatively, move overview text into the docstrings and "just" use `autoapi` for everything. - [x] Use `:py:func:`, `:py:class:`, `:py:meth:`, `:py:mod:`, etc. to cross reference functions and classes in the docstrings. - [x] Add `napolean` to parse `numpydoc` style docstrings (previous `numpydoc` method was incomplete) - [x] Use `intersphinx` and `nitpick` to check (or ignore where not possible) all internal and most external cross references to ensure good code navigation - [x] Fixup all broken references by either adding fully qualified types in docstrings or in the imports. - [x] Ignore type aliase for for now (documented bug, will address in a future PR) - [ ] Add a few high level examples in docstrings. - [ ] May need `from_config_string` style loaders to help with this. - [x] Add `doctest` to `pytest` to validate examples. - [x] Update `mlos_bench.run` help usage output to an `rst` file and link that into the man table of contents tree in the output. --- ## Type of Change - 🛠️ Bug fix - 📝 Documentation update - 🧪 Tests --- ## Testing - Adds additional checks and linting. - Must pass CI - Manual testing as follows: ```sh # SKIP_COVERAGE tweak is optional - just avoids a `pytest` job dependency make SKIP_COVERAGE=true doc ./doc/nginx-docker.sh restart ``` Browse to `http://localhost:8080` and check the results. --- ## Additional Notes Future PRs can add additional documentation strings for the `mlos_bench` classes including examples. In particular - [ ] mlos_bench.optimizers - [ ] mlos_bench.services --------- Co-authored-by: Sergiy Matusevych <sergiy.matusevych@gmail.com>
2024-11-15 21:59:31 +03:00
conda run -n ${CONDA_ENV_NAME} $(MAKE) SPHINXOPTS="$(SPHINXOPTS)" -C doc/ $(MAKEFLAGS) html \
Merged PR 903: Combination of additional build and doc fixups This PR was originally aimed at just fixing the intermittent documentation build errors. That issue ultimately stems from conflicts between conda and pip installs of two different versions of `mistune` dependency. Along the way I found a handful of other improvements for build, pipeline, and documentation generation things. - Make sure we do the nightly build regardless of whether code has changed - Don't build the docs in parallel by default (trying to avoid an intermittent build failure) - Add basic support for testing devcontainer manually outside for vscode - Stop doing parallel builds so it's easier to debug, and move doc build/publishing to the devcontainer - Split windows conda env off, stop using gpy from conda-forge, and just build it locally with clang, gcc, or msvc - Adjust the devcontainer build so that more dependencies are built upfront during the container build. This makes the process more cacheable so that we can just pull the upstream version and not have to build as much locally which could delay first launch. - Import part of a fix from elsewhere - Switch to docker-compose style devcontainer that automatically also starts an nginx container for viewing the generated docs - Update some docs and doc building - [X] pydocstyle fixups - [X] Add some log filtering of the `make doc` output for easier debugging (e.g. don't care about missing examples for now) - [X] doc link lint checks - DONE - [X] Split this out into multiple PRs - [X] SpaceAdapterFactory: !959 - [X] docstyle fixups: !960 - [ ] doc build fixups - [ ] devcontainer improvements (leaving the last two as this PR)
2023-02-27 08:42:22 +03:00
>> doc/build/log.txt 2>&1 \
|| { cat doc/build/log.txt; exit 1; }
# DONE: Add some output filtering for this so we can more easily see what went wrong.
# (e.g. skip complaints about missing .examples files)
# See check-doc
.PHONY: doc
Pypi packaging (#626) Co-authored-by: Sergiy Matusevych <sergiy.matusevych@gmail.com>
2024-01-10 01:40:15 +03:00
doc: doc/build/html/.nojekyll doc-test
.PHONY: doc-test
doc-test: build/check-doc.build-stamp build/linklint-doc.build-stamp
Merged PR 903: Combination of additional build and doc fixups This PR was originally aimed at just fixing the intermittent documentation build errors. That issue ultimately stems from conflicts between conda and pip installs of two different versions of `mistune` dependency. Along the way I found a handful of other improvements for build, pipeline, and documentation generation things. - Make sure we do the nightly build regardless of whether code has changed - Don't build the docs in parallel by default (trying to avoid an intermittent build failure) - Add basic support for testing devcontainer manually outside for vscode - Stop doing parallel builds so it's easier to debug, and move doc build/publishing to the devcontainer - Split windows conda env off, stop using gpy from conda-forge, and just build it locally with clang, gcc, or msvc - Adjust the devcontainer build so that more dependencies are built upfront during the container build. This makes the process more cacheable so that we can just pull the upstream version and not have to build as much locally which could delay first launch. - Import part of a fix from elsewhere - Switch to docker-compose style devcontainer that automatically also starts an nginx container for viewing the generated docs - Update some docs and doc building - [X] pydocstyle fixups - [X] Add some log filtering of the `make doc` output for easier debugging (e.g. don't care about missing examples for now) - [X] doc link lint checks - DONE - [X] Split this out into multiple PRs - [X] SpaceAdapterFactory: !959 - [X] docstyle fixups: !960 - [ ] doc build fixups - [ ] devcontainer improvements (leaving the last two as this PR)
2023-02-27 08:42:22 +03:00
Merged PR 1026: Split test and check rules and improve test coverage publishing in pipelines and docs - make test and check targets for mlos_core and mlos_bench independent for faster execution and more clear output for `make test` and `make check` iteration when only hacking on one module locally - ensures that the full set still runs in the pipeline though - add support for skipping code coverage locally by setting `SKIP_COVERAGE=true` to run tests faster - optionally copy the code coverage data into our static html site for publishing (plus nice badges) - also, rather than racing and publishing the test results and code coverage for all of the different sub jobs, we only run and upload coverage via `pytest-azurepipelines` for the main one we're interested in to make reporting easier - also includes which job the test came from in the results - some other general `Makefile` cleanup Future work: - [X] Split the single pipeline into two pipelines and make the Windows one optional (since it's usually the one that takes the longest and we care a little bit less about it now that we have decontainer support) !1040
2023-03-16 22:47:01 +03:00
doc/build/html/htmlcov/index.html: doc/build/html/index.html
# Make the codecov html report available for the site.
test -d htmlcov && cp -a htmlcov doc/build/html/ || true
mkdir -p doc/build/html/htmlcov
touch doc/build/html/htmlcov/index.html
Switch to Github Actions for CI/CD runs (#274)
2023-03-18 23:52:26 +03:00
doc/build/html/.nojekyll: doc/build/html/index.html doc/build/html/htmlcov/index.html
# Make sure that github pages doesn't try to run jekyll on the docs.
touch doc/build/html/.nojekyll
Merged PR 903: Combination of additional build and doc fixups This PR was originally aimed at just fixing the intermittent documentation build errors. That issue ultimately stems from conflicts between conda and pip installs of two different versions of `mistune` dependency. Along the way I found a handful of other improvements for build, pipeline, and documentation generation things. - Make sure we do the nightly build regardless of whether code has changed - Don't build the docs in parallel by default (trying to avoid an intermittent build failure) - Add basic support for testing devcontainer manually outside for vscode - Stop doing parallel builds so it's easier to debug, and move doc build/publishing to the devcontainer - Split windows conda env off, stop using gpy from conda-forge, and just build it locally with clang, gcc, or msvc - Adjust the devcontainer build so that more dependencies are built upfront during the container build. This makes the process more cacheable so that we can just pull the upstream version and not have to build as much locally which could delay first launch. - Import part of a fix from elsewhere - Switch to docker-compose style devcontainer that automatically also starts an nginx container for viewing the generated docs - Update some docs and doc building - [X] pydocstyle fixups - [X] Add some log filtering of the `make doc` output for easier debugging (e.g. don't care about missing examples for now) - [X] doc link lint checks - DONE - [X] Split this out into multiple PRs - [X] SpaceAdapterFactory: !959 - [X] docstyle fixups: !960 - [ ] doc build fixups - [ ] devcontainer improvements (leaving the last two as this PR)
2023-02-27 08:42:22 +03:00
.PHONY: check-doc
check-doc: build/check-doc.build-stamp
Merged PR 1026: Split test and check rules and improve test coverage publishing in pipelines and docs - make test and check targets for mlos_core and mlos_bench independent for faster execution and more clear output for `make test` and `make check` iteration when only hacking on one module locally - ensures that the full set still runs in the pipeline though - add support for skipping code coverage locally by setting `SKIP_COVERAGE=true` to run tests faster - optionally copy the code coverage data into our static html site for publishing (plus nice badges) - also, rather than racing and publishing the test results and code coverage for all of the different sub jobs, we only run and upload coverage via `pytest-azurepipelines` for the main one we're interested in to make reporting easier - also includes which job the test came from in the results - some other general `Makefile` cleanup Future work: - [X] Split the single pipeline into two pipelines and make the Windows one optional (since it's usually the one that takes the longest and we care a little bit less about it now that we have decontainer support) !1040
2023-03-16 22:47:01 +03:00
build/check-doc.build-stamp: doc/build/html/index.html doc/build/html/htmlcov/index.html
Merged PR 903: Combination of additional build and doc fixups This PR was originally aimed at just fixing the intermittent documentation build errors. That issue ultimately stems from conflicts between conda and pip installs of two different versions of `mistune` dependency. Along the way I found a handful of other improvements for build, pipeline, and documentation generation things. - Make sure we do the nightly build regardless of whether code has changed - Don't build the docs in parallel by default (trying to avoid an intermittent build failure) - Add basic support for testing devcontainer manually outside for vscode - Stop doing parallel builds so it's easier to debug, and move doc build/publishing to the devcontainer - Split windows conda env off, stop using gpy from conda-forge, and just build it locally with clang, gcc, or msvc - Adjust the devcontainer build so that more dependencies are built upfront during the container build. This makes the process more cacheable so that we can just pull the upstream version and not have to build as much locally which could delay first launch. - Import part of a fix from elsewhere - Switch to docker-compose style devcontainer that automatically also starts an nginx container for viewing the generated docs - Update some docs and doc building - [X] pydocstyle fixups - [X] Add some log filtering of the `make doc` output for easier debugging (e.g. don't care about missing examples for now) - [X] doc link lint checks - DONE - [X] Split this out into multiple PRs - [X] SpaceAdapterFactory: !959 - [X] docstyle fixups: !960 - [ ] doc build fixups - [ ] devcontainer improvements (leaving the last two as this PR)
2023-02-27 08:42:22 +03:00
# Check for a few files to make sure the docs got generated in a way we want.
Merged PR 386: Generate and publish documentation from python docstrings - initial attempts at generating documentation Related work items: #307
2022-03-17 04:33:28 +03:00
test -s doc/build/html/index.html
Rework documentation generation (#869) # Pull Request ## Title Documentation generation rework. --- ## Description This PR does an initial revamp of the way we generate and write documentation. The goal is to simplify the generation process a bit by removing some hacks but also enable more strict checking of the documentation produced so that links resolve properly so that it is easier to navigate. This PR is mostly the mechanics and the associated fixups to do that. Future PRs can handle additional documentation in docstrings. Below is a more detailed description of some of the changes. - [x] Apply `black` and `pylint` to the sphinx `conf.py` - [x] Treat sphinx warnings as errors - [x] Reorg doc generation using `autoapi` to handle cross referencing warnings and drop use of `sphinx-apidoc` - [x] ~Update `overview.rst` to follow suite~ - [x] Alternatively, move overview text into the docstrings and "just" use `autoapi` for everything. - [x] Use `:py:func:`, `:py:class:`, `:py:meth:`, `:py:mod:`, etc. to cross reference functions and classes in the docstrings. - [x] Add `napolean` to parse `numpydoc` style docstrings (previous `numpydoc` method was incomplete) - [x] Use `intersphinx` and `nitpick` to check (or ignore where not possible) all internal and most external cross references to ensure good code navigation - [x] Fixup all broken references by either adding fully qualified types in docstrings or in the imports. - [x] Ignore type aliase for for now (documented bug, will address in a future PR) - [ ] Add a few high level examples in docstrings. - [ ] May need `from_config_string` style loaders to help with this. - [x] Add `doctest` to `pytest` to validate examples. - [x] Update `mlos_bench.run` help usage output to an `rst` file and link that into the man table of contents tree in the output. --- ## Type of Change - 🛠️ Bug fix - 📝 Documentation update - 🧪 Tests --- ## Testing - Adds additional checks and linting. - Must pass CI - Manual testing as follows: ```sh # SKIP_COVERAGE tweak is optional - just avoids a `pytest` job dependency make SKIP_COVERAGE=true doc ./doc/nginx-docker.sh restart ``` Browse to `http://localhost:8080` and check the results. --- ## Additional Notes Future PRs can add additional documentation strings for the `mlos_bench` classes including examples. In particular - [ ] mlos_bench.optimizers - [ ] mlos_bench.services --------- Co-authored-by: Sergiy Matusevych <sergiy.matusevych@gmail.com>
2024-11-15 21:59:31 +03:00
grep -q BaseOptimizer doc/build/html/autoapi/mlos_core/optimizers/optimizer/index.html
grep -q Environment doc/build/html/autoapi/mlos_bench/environments/base_environment/index.html
grep -q plot doc/build/html/autoapi/mlos_viz/index.html
test -s doc/build/html/autoapi/mlos_core/index.html
test -s doc/build/html/autoapi/mlos_bench/index.html
test -s doc/build/html/autoapi/mlos_viz/index.html
test -s doc/build/html/autoapi/mlos_viz/dabl/index.html
grep -q -e '--config CONFIG' doc/build/html//mlos_bench.run.usage.html
Merged PR 903: Combination of additional build and doc fixups This PR was originally aimed at just fixing the intermittent documentation build errors. That issue ultimately stems from conflicts between conda and pip installs of two different versions of `mistune` dependency. Along the way I found a handful of other improvements for build, pipeline, and documentation generation things. - Make sure we do the nightly build regardless of whether code has changed - Don't build the docs in parallel by default (trying to avoid an intermittent build failure) - Add basic support for testing devcontainer manually outside for vscode - Stop doing parallel builds so it's easier to debug, and move doc build/publishing to the devcontainer - Split windows conda env off, stop using gpy from conda-forge, and just build it locally with clang, gcc, or msvc - Adjust the devcontainer build so that more dependencies are built upfront during the container build. This makes the process more cacheable so that we can just pull the upstream version and not have to build as much locally which could delay first launch. - Import part of a fix from elsewhere - Switch to docker-compose style devcontainer that automatically also starts an nginx container for viewing the generated docs - Update some docs and doc building - [X] pydocstyle fixups - [X] Add some log filtering of the `make doc` output for easier debugging (e.g. don't care about missing examples for now) - [X] doc link lint checks - DONE - [X] Split this out into multiple PRs - [X] SpaceAdapterFactory: !959 - [X] docstyle fixups: !960 - [ ] doc build fixups - [ ] devcontainer improvements (leaving the last two as this PR)
2023-02-27 08:42:22 +03:00
# Check doc logs for errors (but skip over some known ones) ...
@cat doc/build/log.txt \
| egrep -C1 -e WARNING -e CRITICAL -e ERROR \
| egrep -v \
Automatic versioning support (#278) * adding configs to be able to use bumpversion for managing versions * Bump version: 0.0.4 → 0.1.0 * cleanup old file * don't need to reinstall licenseheaders everytime if we include it in the conda env * github actions checks out in shallow mode which setuptools-scm doesn't like, but doesn't cause issues, so we ignore the warning
2023-03-21 01:46:06 +03:00
-e "warnings.warn\(f'\"{wd.path}\" is shallow and may cause errors'\)" \
Various quick CI fixups (#870) # Pull Request ## Title Quick fixups to the documentation build. --- ## Description - Expands one of the ignored warnings. - Reformats the sphinx conf for black and pylint. - Removes a warning from that file around redefinition of `html_theme_path`. - Adds a dependency requirement tweak for `pyparsing` when installing under python 3.8 (unrelated other CI bug). - Adds a workaround to a mypy false positive with in checking `np.e` as a "deleted variable" (e.g., `e` is used in a try/except block) --- ## Type of Change - 🛠️ Bug fix - 📝 Documentation update --- ## Testing - Ran `make doc` and `make check-doc`. - Ran `make notebook-exec-test`. --- ## Additional Notes (optional) This is a quick fix to get the CI pipeline working again. The more complete fix to remove warning ignores from doc generation, improve cross reference linking, etc. will be handled later in #869 ---
2024-10-15 00:25:10 +03:00
-e "No such file or directory: '.*.examples'.( \[docutils\]\s*)?$$" \
Merged PR 903: Combination of additional build and doc fixups This PR was originally aimed at just fixing the intermittent documentation build errors. That issue ultimately stems from conflicts between conda and pip installs of two different versions of `mistune` dependency. Along the way I found a handful of other improvements for build, pipeline, and documentation generation things. - Make sure we do the nightly build regardless of whether code has changed - Don't build the docs in parallel by default (trying to avoid an intermittent build failure) - Add basic support for testing devcontainer manually outside for vscode - Stop doing parallel builds so it's easier to debug, and move doc build/publishing to the devcontainer - Split windows conda env off, stop using gpy from conda-forge, and just build it locally with clang, gcc, or msvc - Adjust the devcontainer build so that more dependencies are built upfront during the container build. This makes the process more cacheable so that we can just pull the upstream version and not have to build as much locally which could delay first launch. - Import part of a fix from elsewhere - Switch to docker-compose style devcontainer that automatically also starts an nginx container for viewing the generated docs - Update some docs and doc building - [X] pydocstyle fixups - [X] Add some log filtering of the `make doc` output for easier debugging (e.g. don't care about missing examples for now) - [X] doc link lint checks - DONE - [X] Split this out into multiple PRs - [X] SpaceAdapterFactory: !959 - [X] docstyle fixups: !960 - [ ] doc build fixups - [ ] devcontainer improvements (leaving the last two as this PR)
2023-02-27 08:42:22 +03:00
-e "toctree contains reference to nonexisting document 'auto_examples/index'" \
-e '^make.*resetting jobserver mode' \
make doc CI fixups (#830) Splitting out from #802 to benefit other jobs as well.
2024-08-01 23:00:00 +03:00
-e 'from cryptography.hazmat.primitives.ciphers.algorithms import' \
Merged PR 903: Combination of additional build and doc fixups This PR was originally aimed at just fixing the intermittent documentation build errors. That issue ultimately stems from conflicts between conda and pip installs of two different versions of `mistune` dependency. Along the way I found a handful of other improvements for build, pipeline, and documentation generation things. - Make sure we do the nightly build regardless of whether code has changed - Don't build the docs in parallel by default (trying to avoid an intermittent build failure) - Add basic support for testing devcontainer manually outside for vscode - Stop doing parallel builds so it's easier to debug, and move doc build/publishing to the devcontainer - Split windows conda env off, stop using gpy from conda-forge, and just build it locally with clang, gcc, or msvc - Adjust the devcontainer build so that more dependencies are built upfront during the container build. This makes the process more cacheable so that we can just pull the upstream version and not have to build as much locally which could delay first launch. - Import part of a fix from elsewhere - Switch to docker-compose style devcontainer that automatically also starts an nginx container for viewing the generated docs - Update some docs and doc building - [X] pydocstyle fixups - [X] Add some log filtering of the `make doc` output for easier debugging (e.g. don't care about missing examples for now) - [X] doc link lint checks - DONE - [X] Split this out into multiple PRs - [X] SpaceAdapterFactory: !959 - [X] docstyle fixups: !960 - [ ] doc build fixups - [ ] devcontainer improvements (leaving the last two as this PR)
2023-02-27 08:42:22 +03:00
| grep -v '^\s*$$' \
make doc CI fixups (#830) Splitting out from #802 to benefit other jobs as well.
2024-08-01 23:00:00 +03:00
| if grep .; then echo "Errors found in doc build. Check doc/build/log.txt for details."; cat doc/build/log.txt; exit 1; else exit 0; fi
Merged PR 1026: Split test and check rules and improve test coverage publishing in pipelines and docs - make test and check targets for mlos_core and mlos_bench independent for faster execution and more clear output for `make test` and `make check` iteration when only hacking on one module locally - ensures that the full set still runs in the pipeline though - add support for skipping code coverage locally by setting `SKIP_COVERAGE=true` to run tests faster - optionally copy the code coverage data into our static html site for publishing (plus nice badges) - also, rather than racing and publishing the test results and code coverage for all of the different sub jobs, we only run and upload coverage via `pytest-azurepipelines` for the main one we're interested in to make reporting easier - also includes which job the test came from in the results - some other general `Makefile` cleanup Future work: - [X] Split the single pipeline into two pipelines and make the Windows one optional (since it's usually the one that takes the longest and we care a little bit less about it now that we have decontainer support) !1040
2023-03-16 22:47:01 +03:00
touch $@
Merged PR 903: Combination of additional build and doc fixups This PR was originally aimed at just fixing the intermittent documentation build errors. That issue ultimately stems from conflicts between conda and pip installs of two different versions of `mistune` dependency. Along the way I found a handful of other improvements for build, pipeline, and documentation generation things. - Make sure we do the nightly build regardless of whether code has changed - Don't build the docs in parallel by default (trying to avoid an intermittent build failure) - Add basic support for testing devcontainer manually outside for vscode - Stop doing parallel builds so it's easier to debug, and move doc build/publishing to the devcontainer - Split windows conda env off, stop using gpy from conda-forge, and just build it locally with clang, gcc, or msvc - Adjust the devcontainer build so that more dependencies are built upfront during the container build. This makes the process more cacheable so that we can just pull the upstream version and not have to build as much locally which could delay first launch. - Import part of a fix from elsewhere - Switch to docker-compose style devcontainer that automatically also starts an nginx container for viewing the generated docs - Update some docs and doc building - [X] pydocstyle fixups - [X] Add some log filtering of the `make doc` output for easier debugging (e.g. don't care about missing examples for now) - [X] doc link lint checks - DONE - [X] Split this out into multiple PRs - [X] SpaceAdapterFactory: !959 - [X] docstyle fixups: !960 - [ ] doc build fixups - [ ] devcontainer improvements (leaving the last two as this PR)
2023-02-27 08:42:22 +03:00
.PHONY: linklint-doc
ifeq ($(DOCKER),)
linklint-doc:
@echo "NOTE: linklint-doc target requires docker."
else
linklint-doc: build/linklint-doc.build-stamp
endif
Doc linklint fixups (#894) # Pull Request ## Title Doc linklint fixups --- ## Description GithubPages serves our docs out of a `/MLOS` path, but previously we checked things out of `/` using an Nginx docker container. This PR changes the Nginx docker container used for testing to also serve paths out of `/MLOS` and fixes the broken links found from that. - [x] Adjust Nginx to serve from `/MLOS` like Github pages. - [x] Fix linklint errors found from that - [x] spelling - See #891 --- ## Type of Change - 📝 Documentation update --- ## Testing - Existing CI (which includes docstring and some link lint checks) ---
2024-12-07 23:08:18 +03:00
.PHONY: nginx_port_env
nginx_port_env:
Merged PR 903: Combination of additional build and doc fixups This PR was originally aimed at just fixing the intermittent documentation build errors. That issue ultimately stems from conflicts between conda and pip installs of two different versions of `mistune` dependency. Along the way I found a handful of other improvements for build, pipeline, and documentation generation things. - Make sure we do the nightly build regardless of whether code has changed - Don't build the docs in parallel by default (trying to avoid an intermittent build failure) - Add basic support for testing devcontainer manually outside for vscode - Stop doing parallel builds so it's easier to debug, and move doc build/publishing to the devcontainer - Split windows conda env off, stop using gpy from conda-forge, and just build it locally with clang, gcc, or msvc - Adjust the devcontainer build so that more dependencies are built upfront during the container build. This makes the process more cacheable so that we can just pull the upstream version and not have to build as much locally which could delay first launch. - Import part of a fix from elsewhere - Switch to docker-compose style devcontainer that automatically also starts an nginx container for viewing the generated docs - Update some docs and doc building - [X] pydocstyle fixups - [X] Add some log filtering of the `make doc` output for easier debugging (e.g. don't care about missing examples for now) - [X] doc link lint checks - DONE - [X] Split this out into multiple PRs - [X] SpaceAdapterFactory: !959 - [X] docstyle fixups: !960 - [ ] doc build fixups - [ ] devcontainer improvements (leaving the last two as this PR)
2023-02-27 08:42:22 +03:00
@echo "Starting nginx docker container for serving docs."
Small tweaks (#901) # Pull Request ## Title Small tweaks --- ## Description - mypy fixup - Minor perf improvement on Makefile file enumeration. - Add typevar hints for `nullable` return type. - Fix a minor build system issue - List python 3.13 as one of the supported versions (#879) --- ## Type of Change - 🛠️ Bug fix - 🔄 Refactor --- ## Testing Usual pytest and mypy CI checks for `nullable` --- ## Additional Notes (optional) Split out from #720 ---
2025-01-04 00:55:27 +03:00
mkdir -p doc/build
Merged PR 903: Combination of additional build and doc fixups This PR was originally aimed at just fixing the intermittent documentation build errors. That issue ultimately stems from conflicts between conda and pip installs of two different versions of `mistune` dependency. Along the way I found a handful of other improvements for build, pipeline, and documentation generation things. - Make sure we do the nightly build regardless of whether code has changed - Don't build the docs in parallel by default (trying to avoid an intermittent build failure) - Add basic support for testing devcontainer manually outside for vscode - Stop doing parallel builds so it's easier to debug, and move doc build/publishing to the devcontainer - Split windows conda env off, stop using gpy from conda-forge, and just build it locally with clang, gcc, or msvc - Adjust the devcontainer build so that more dependencies are built upfront during the container build. This makes the process more cacheable so that we can just pull the upstream version and not have to build as much locally which could delay first launch. - Import part of a fix from elsewhere - Switch to docker-compose style devcontainer that automatically also starts an nginx container for viewing the generated docs - Update some docs and doc building - [X] pydocstyle fixups - [X] Add some log filtering of the `make doc` output for easier debugging (e.g. don't care about missing examples for now) - [X] doc link lint checks - DONE - [X] Split this out into multiple PRs - [X] SpaceAdapterFactory: !959 - [X] docstyle fixups: !960 - [ ] doc build fixups - [ ] devcontainer improvements (leaving the last two as this PR)
2023-02-27 08:42:22 +03:00
./doc/nginx-docker.sh restart
devcontainer nginx port fixups (#307) * move nginx port to a random port * make it work for powershell too * need the .env file to be in a different directory * fixups for pwsh * also stop running the docker pull * whitespace reorg * fixup * make the cmd more verbose * more debugging * ignore ipv6
2023-04-17 23:11:29 +03:00
nginx_port=`docker port mlos-doc-nginx | grep 0.0.0.0:8080 | cut -d/ -f1` \
Doc linklint fixups (#894) # Pull Request ## Title Doc linklint fixups --- ## Description GithubPages serves our docs out of a `/MLOS` path, but previously we checked things out of `/` using an Nginx docker container. This PR changes the Nginx docker container used for testing to also serve paths out of `/MLOS` and fixes the broken links found from that. - [x] Adjust Nginx to serve from `/MLOS` like Github pages. - [x] Fix linklint errors found from that - [x] spelling - See #891 --- ## Type of Change - 📝 Documentation update --- ## Testing - Existing CI (which includes docstring and some link lint checks) ---
2024-12-07 23:08:18 +03:00
&& echo nginx_port=$${nginx_port} > doc/build/nginx_port.env
build/linklint-doc.build-stamp: nginx_port=$(shell cat doc/build/nginx_port.env | cut -d= -f2 | egrep -x '[0-9]+')
build/linklint-doc.build-stamp: doc/build/html/index.html doc/build/html/htmlcov/index.html build/check-doc.build-stamp nginx_port_env
@echo "Running linklint on the docs at http://localhost:${nginx_port}/MLOS/ ..."
docker exec mlos-doc-nginx curl -sSf http://localhost:${nginx_port}/MLOS/index.html >/dev/null
docker exec mlos-doc-nginx linklint -root /doc/build/html/ /@ -error -warn > doc/build/linklint.out 2>&1
docker exec mlos-doc-nginx linklint -net -redirect -host localhost:${nginx_port} /MLOS/@ -http -error -warn > doc/build/linklint.out 2>&1
Merged PR 903: Combination of additional build and doc fixups This PR was originally aimed at just fixing the intermittent documentation build errors. That issue ultimately stems from conflicts between conda and pip installs of two different versions of `mistune` dependency. Along the way I found a handful of other improvements for build, pipeline, and documentation generation things. - Make sure we do the nightly build regardless of whether code has changed - Don't build the docs in parallel by default (trying to avoid an intermittent build failure) - Add basic support for testing devcontainer manually outside for vscode - Stop doing parallel builds so it's easier to debug, and move doc build/publishing to the devcontainer - Split windows conda env off, stop using gpy from conda-forge, and just build it locally with clang, gcc, or msvc - Adjust the devcontainer build so that more dependencies are built upfront during the container build. This makes the process more cacheable so that we can just pull the upstream version and not have to build as much locally which could delay first launch. - Import part of a fix from elsewhere - Switch to docker-compose style devcontainer that automatically also starts an nginx container for viewing the generated docs - Update some docs and doc building - [X] pydocstyle fixups - [X] Add some log filtering of the `make doc` output for easier debugging (e.g. don't care about missing examples for now) - [X] doc link lint checks - DONE - [X] Split this out into multiple PRs - [X] SpaceAdapterFactory: !959 - [X] docstyle fixups: !960 - [ ] doc build fixups - [ ] devcontainer improvements (leaving the last two as this PR)
2023-02-27 08:42:22 +03:00
@if cat doc/build/linklint.out | grep -e ^ERROR -e ^WARN | grep -v 'missing named anchors' | grep -q .; then \
echo "Found errors in the documentation:"; cat doc/build/linklint.out; exit 1; \
fi
@echo "OK"
Merged PR 1026: Split test and check rules and improve test coverage publishing in pipelines and docs - make test and check targets for mlos_core and mlos_bench independent for faster execution and more clear output for `make test` and `make check` iteration when only hacking on one module locally - ensures that the full set still runs in the pipeline though - add support for skipping code coverage locally by setting `SKIP_COVERAGE=true` to run tests faster - optionally copy the code coverage data into our static html site for publishing (plus nice badges) - also, rather than racing and publishing the test results and code coverage for all of the different sub jobs, we only run and upload coverage via `pytest-azurepipelines` for the main one we're interested in to make reporting easier - also includes which job the test came from in the results - some other general `Makefile` cleanup Future work: - [X] Split the single pipeline into two pipelines and make the Windows one optional (since it's usually the one that takes the longest and we care a little bit less about it now that we have decontainer support) !1040
2023-03-16 22:47:01 +03:00
touch $@
Merged PR 386: Generate and publish documentation from python docstrings - initial attempts at generating documentation Related work items: #307
2022-03-17 04:33:28 +03:00
Staged Makefile Improvements (#762) - Introduce a new `make format` target that currently only calls `licenseheaders` but will call `isort` and `black` in the future - Introduce new variables to help make dependency tracking easier - Simplify `pytest` rules and allow running only a single module at a time. This PR is a precursor to pyproject.toml related build changes as well. --------- Co-authored-by: Sergiy Matusevych <sergiym@microsoft.com>
2024-07-01 18:57:17 +03:00
Merged PR 386: Generate and publish documentation from python docstrings - initial attempts at generating documentation Related work items: #307
2022-03-17 04:33:28 +03:00
.PHONY: clean-doc
clean-doc:
Rework documentation generation (#869) # Pull Request ## Title Documentation generation rework. --- ## Description This PR does an initial revamp of the way we generate and write documentation. The goal is to simplify the generation process a bit by removing some hacks but also enable more strict checking of the documentation produced so that links resolve properly so that it is easier to navigate. This PR is mostly the mechanics and the associated fixups to do that. Future PRs can handle additional documentation in docstrings. Below is a more detailed description of some of the changes. - [x] Apply `black` and `pylint` to the sphinx `conf.py` - [x] Treat sphinx warnings as errors - [x] Reorg doc generation using `autoapi` to handle cross referencing warnings and drop use of `sphinx-apidoc` - [x] ~Update `overview.rst` to follow suite~ - [x] Alternatively, move overview text into the docstrings and "just" use `autoapi` for everything. - [x] Use `:py:func:`, `:py:class:`, `:py:meth:`, `:py:mod:`, etc. to cross reference functions and classes in the docstrings. - [x] Add `napolean` to parse `numpydoc` style docstrings (previous `numpydoc` method was incomplete) - [x] Use `intersphinx` and `nitpick` to check (or ignore where not possible) all internal and most external cross references to ensure good code navigation - [x] Fixup all broken references by either adding fully qualified types in docstrings or in the imports. - [x] Ignore type aliase for for now (documented bug, will address in a future PR) - [ ] Add a few high level examples in docstrings. - [ ] May need `from_config_string` style loaders to help with this. - [x] Add `doctest` to `pytest` to validate examples. - [x] Update `mlos_bench.run` help usage output to an `rst` file and link that into the man table of contents tree in the output. --- ## Type of Change - 🛠️ Bug fix - 📝 Documentation update - 🧪 Tests --- ## Testing - Adds additional checks and linting. - Must pass CI - Manual testing as follows: ```sh # SKIP_COVERAGE tweak is optional - just avoids a `pytest` job dependency make SKIP_COVERAGE=true doc ./doc/nginx-docker.sh restart ``` Browse to `http://localhost:8080` and check the results. --- ## Additional Notes Future PRs can add additional documentation strings for the `mlos_bench` classes including examples. In particular - [ ] mlos_bench.optimizers - [ ] mlos_bench.services --------- Co-authored-by: Sergiy Matusevych <sergiy.matusevych@gmail.com>
2024-11-15 21:59:31 +03:00
rm -rf doc/build/ doc/global/ doc/source/api/ doc/source/generated doc/source/autoapi
experiment_id documentation improvements (#601) Closes #599
2023-11-20 23:50:49 +03:00
rm -rf doc/source/source_tree_docs/*
Makefile bdist rule tweaks (#316) Followup cleanup to #315
2023-04-26 02:46:16 +03:00
Staged Makefile Improvements (#762) - Introduce a new `make format` target that currently only calls `licenseheaders` but will call `isort` and `black` in the future - Introduce new variables to help make dependency tracking easier - Simplify `pytest` rules and allow running only a single module at a time. This PR is a precursor to pyproject.toml related build changes as well. --------- Co-authored-by: Sergiy Matusevych <sergiym@microsoft.com>
2024-07-01 18:57:17 +03:00
.PHONY: clean-format
clean-format:
rm -f build/black.${CONDA_ENV_NAME}.build-stamp
Introduce basic experiment visualization module mlos_viz via dabl (#624) Adds a basic `mlos_viz.plot(exp)` style API for simple visualizations of `ExperimentData` results relative to the experiment's objectives (building off of #628 and https://github.com/dabl/dabl/pull/335). Note: this PR currently omits unit tests for the new module due to the complexity of testing visualizations. We intend to add this in future PRs. There is however, a working example of its use here right now: https://github.com/Microsoft-CISL/sqlite-autotuning/pull/41 --------- Co-authored-by: Sergiy Matusevych <sergiy.matusevych@gmail.com>
2024-01-16 21:48:20 +03:00
rm -f build/black.mlos_*.${CONDA_ENV_NAME}.build-stamp
Staged Makefile Improvements (#762) - Introduce a new `make format` target that currently only calls `licenseheaders` but will call `isort` and `black` in the future - Introduce new variables to help make dependency tracking easier - Simplify `pytest` rules and allow running only a single module at a time. This PR is a precursor to pyproject.toml related build changes as well. --------- Co-authored-by: Sergiy Matusevych <sergiym@microsoft.com>
2024-07-01 18:57:17 +03:00
rm -f build/docformatter.${CONDA_ENV_NAME}.build-stamp
Introduce basic experiment visualization module mlos_viz via dabl (#624) Adds a basic `mlos_viz.plot(exp)` style API for simple visualizations of `ExperimentData` results relative to the experiment's objectives (building off of #628 and https://github.com/dabl/dabl/pull/335). Note: this PR currently omits unit tests for the new module due to the complexity of testing visualizations. We intend to add this in future PRs. There is however, a working example of its use here right now: https://github.com/Microsoft-CISL/sqlite-autotuning/pull/41 --------- Co-authored-by: Sergiy Matusevych <sergiy.matusevych@gmail.com>
2024-01-16 21:48:20 +03:00
rm -f build/docformatter.mlos_*.${CONDA_ENV_NAME}.build-stamp
Staged Makefile Improvements (#762) - Introduce a new `make format` target that currently only calls `licenseheaders` but will call `isort` and `black` in the future - Introduce new variables to help make dependency tracking easier - Simplify `pytest` rules and allow running only a single module at a time. This PR is a precursor to pyproject.toml related build changes as well. --------- Co-authored-by: Sergiy Matusevych <sergiym@microsoft.com>
2024-07-01 18:57:17 +03:00
rm -f build/isort.${CONDA_ENV_NAME}.build-stamp
Introduce basic experiment visualization module mlos_viz via dabl (#624) Adds a basic `mlos_viz.plot(exp)` style API for simple visualizations of `ExperimentData` results relative to the experiment's objectives (building off of #628 and https://github.com/dabl/dabl/pull/335). Note: this PR currently omits unit tests for the new module due to the complexity of testing visualizations. We intend to add this in future PRs. There is however, a working example of its use here right now: https://github.com/Microsoft-CISL/sqlite-autotuning/pull/41 --------- Co-authored-by: Sergiy Matusevych <sergiy.matusevych@gmail.com>
2024-01-16 21:48:20 +03:00
rm -f build/isort.mlos_*.${CONDA_ENV_NAME}.build-stamp
Staged Makefile Improvements (#762) - Introduce a new `make format` target that currently only calls `licenseheaders` but will call `isort` and `black` in the future - Introduce new variables to help make dependency tracking easier - Simplify `pytest` rules and allow running only a single module at a time. This PR is a precursor to pyproject.toml related build changes as well. --------- Co-authored-by: Sergiy Matusevych <sergiym@microsoft.com>
2024-07-01 18:57:17 +03:00
rm -f build/licenseheaders.${CONDA_ENV_NAME}.build-stamp
rm -f build/licenseheaders-prereqs.${CONDA_ENV_NAME}.build-stamp
Rework format lint checks for pre-commit (#904) # Pull Request ## Title Reworks build to use pre-commit for format and lint checks. --- ## Description As mentioned in #902 and #903, this refactors the build system to use [`pre-commit`](https://pre-commit.com) for the following reasons: - less complicated Makefile rules - pinned linter versions that can be checked with dependabot more explicitly for easier maintenance - See Also: #749 --- ## Type of Change - ✨ New feature - 🔄 Refactor - 📝 Documentation update - 🧪 Tests --- ## Testing Manually locally and usual CI tests. --- ## Additional Notes (optional) This builds off of #903 and is prep work to change the syntax of the codebase using `pyupgrade`. ---
2025-01-04 01:58:07 +03:00
rm -f build/format.${CONDA_ENV_NAME}.build-stamp
Staged Makefile Improvements (#762) - Introduce a new `make format` target that currently only calls `licenseheaders` but will call `isort` and `black` in the future - Introduce new variables to help make dependency tracking easier - Simplify `pytest` rules and allow running only a single module at a time. This PR is a precursor to pyproject.toml related build changes as well. --------- Co-authored-by: Sergiy Matusevych <sergiym@microsoft.com>
2024-07-01 18:57:17 +03:00
Merged PR 333: Initial MlosCore Implementation Adds basic `register` and `suggest` APIs for a couple of optimizer backends: `Skopt`, `Emukit`, `Random`. Includes example notebook for starters. Limited unit testing. Basic function documentation. Azure DevOps CI pipelines for `pylint` and `pytest` via the `mlos_core` `conda` `environment.yml`. Related work items: #274, #279
2022-03-14 22:37:52 +03:00
.PHONY: clean-check
clean-check:
Merged PR 903: Combination of additional build and doc fixups This PR was originally aimed at just fixing the intermittent documentation build errors. That issue ultimately stems from conflicts between conda and pip installs of two different versions of `mistune` dependency. Along the way I found a handful of other improvements for build, pipeline, and documentation generation things. - Make sure we do the nightly build regardless of whether code has changed - Don't build the docs in parallel by default (trying to avoid an intermittent build failure) - Add basic support for testing devcontainer manually outside for vscode - Stop doing parallel builds so it's easier to debug, and move doc build/publishing to the devcontainer - Split windows conda env off, stop using gpy from conda-forge, and just build it locally with clang, gcc, or msvc - Adjust the devcontainer build so that more dependencies are built upfront during the container build. This makes the process more cacheable so that we can just pull the upstream version and not have to build as much locally which could delay first launch. - Import part of a fix from elsewhere - Switch to docker-compose style devcontainer that automatically also starts an nginx container for viewing the generated docs - Update some docs and doc building - [X] pydocstyle fixups - [X] Add some log filtering of the `make doc` output for easier debugging (e.g. don't care about missing examples for now) - [X] doc link lint checks - DONE - [X] Split this out into multiple PRs - [X] SpaceAdapterFactory: !959 - [X] docstyle fixups: !960 - [ ] doc build fixups - [ ] devcontainer improvements (leaving the last two as this PR)
2023-02-27 08:42:22 +03:00
rm -f build/pylint.build-stamp
rm -f build/pylint.${CONDA_ENV_NAME}.build-stamp
Introduce basic experiment visualization module mlos_viz via dabl (#624) Adds a basic `mlos_viz.plot(exp)` style API for simple visualizations of `ExperimentData` results relative to the experiment's objectives (building off of #628 and https://github.com/dabl/dabl/pull/335). Note: this PR currently omits unit tests for the new module due to the complexity of testing visualizations. We intend to add this in future PRs. There is however, a working example of its use here right now: https://github.com/Microsoft-CISL/sqlite-autotuning/pull/41 --------- Co-authored-by: Sergiy Matusevych <sergiy.matusevych@gmail.com>
2024-01-16 21:48:20 +03:00
rm -f build/pylint.mlos_*.${CONDA_ENV_NAME}.build-stamp
rm -f build/mypy.mlos_*.${CONDA_ENV_NAME}.build-stamp
Prepare isort black formatters and checks (#766) Builds off of #762, #763, and #764. Prepares rules and configs to enable isort and black formatters and checks but doesn't enable them yet. After these are enabled (next PR) we will reformat all files and ignore that revision in git blame configs. Then, we can convert configs stored in `setup.cfg` and `.pylintrc` to the top level `pyproject.toml` and remove the older configs.
2024-07-03 23:57:49 +03:00
rm -f build/black-check.build-stamp
rm -f build/black-check.${CONDA_ENV_NAME}.build-stamp
rm -f build/black-check.mlos_*.${CONDA_ENV_NAME}.build-stamp
Merged PR 903: Combination of additional build and doc fixups This PR was originally aimed at just fixing the intermittent documentation build errors. That issue ultimately stems from conflicts between conda and pip installs of two different versions of `mistune` dependency. Along the way I found a handful of other improvements for build, pipeline, and documentation generation things. - Make sure we do the nightly build regardless of whether code has changed - Don't build the docs in parallel by default (trying to avoid an intermittent build failure) - Add basic support for testing devcontainer manually outside for vscode - Stop doing parallel builds so it's easier to debug, and move doc build/publishing to the devcontainer - Split windows conda env off, stop using gpy from conda-forge, and just build it locally with clang, gcc, or msvc - Adjust the devcontainer build so that more dependencies are built upfront during the container build. This makes the process more cacheable so that we can just pull the upstream version and not have to build as much locally which could delay first launch. - Import part of a fix from elsewhere - Switch to docker-compose style devcontainer that automatically also starts an nginx container for viewing the generated docs - Update some docs and doc building - [X] pydocstyle fixups - [X] Add some log filtering of the `make doc` output for easier debugging (e.g. don't care about missing examples for now) - [X] doc link lint checks - DONE - [X] Split this out into multiple PRs - [X] SpaceAdapterFactory: !959 - [X] docstyle fixups: !960 - [ ] doc build fixups - [ ] devcontainer improvements (leaving the last two as this PR)
2023-02-27 08:42:22 +03:00
rm -f build/docformatter-check.${CONDA_ENV_NAME}.build-stamp
Introduce basic experiment visualization module mlos_viz via dabl (#624) Adds a basic `mlos_viz.plot(exp)` style API for simple visualizations of `ExperimentData` results relative to the experiment's objectives (building off of #628 and https://github.com/dabl/dabl/pull/335). Note: this PR currently omits unit tests for the new module due to the complexity of testing visualizations. We intend to add this in future PRs. There is however, a working example of its use here right now: https://github.com/Microsoft-CISL/sqlite-autotuning/pull/41 --------- Co-authored-by: Sergiy Matusevych <sergiy.matusevych@gmail.com>
2024-01-16 21:48:20 +03:00
rm -f build/docformatter-check.mlos_*.${CONDA_ENV_NAME}.build-stamp
Prepare isort black formatters and checks (#766) Builds off of #762, #763, and #764. Prepares rules and configs to enable isort and black formatters and checks but doesn't enable them yet. After these are enabled (next PR) we will reformat all files and ignore that revision in git blame configs. Then, we can convert configs stored in `setup.cfg` and `.pylintrc` to the top level `pyproject.toml` and remove the older configs.
2024-07-03 23:57:49 +03:00
rm -f build/isort-check.${CONDA_ENV_NAME}.build-stamp
rm -f build/isort-check.mlos_*.${CONDA_ENV_NAME}.build-stamp
Merged PR 903: Combination of additional build and doc fixups This PR was originally aimed at just fixing the intermittent documentation build errors. That issue ultimately stems from conflicts between conda and pip installs of two different versions of `mistune` dependency. Along the way I found a handful of other improvements for build, pipeline, and documentation generation things. - Make sure we do the nightly build regardless of whether code has changed - Don't build the docs in parallel by default (trying to avoid an intermittent build failure) - Add basic support for testing devcontainer manually outside for vscode - Stop doing parallel builds so it's easier to debug, and move doc build/publishing to the devcontainer - Split windows conda env off, stop using gpy from conda-forge, and just build it locally with clang, gcc, or msvc - Adjust the devcontainer build so that more dependencies are built upfront during the container build. This makes the process more cacheable so that we can just pull the upstream version and not have to build as much locally which could delay first launch. - Import part of a fix from elsewhere - Switch to docker-compose style devcontainer that automatically also starts an nginx container for viewing the generated docs - Update some docs and doc building - [X] pydocstyle fixups - [X] Add some log filtering of the `make doc` output for easier debugging (e.g. don't care about missing examples for now) - [X] doc link lint checks - DONE - [X] Split this out into multiple PRs - [X] SpaceAdapterFactory: !959 - [X] docstyle fixups: !960 - [ ] doc build fixups - [ ] devcontainer improvements (leaving the last two as this PR)
2023-02-27 08:42:22 +03:00
rm -f build/pycodestyle.build-stamp
rm -f build/pycodestyle.${CONDA_ENV_NAME}.build-stamp
Introduce basic experiment visualization module mlos_viz via dabl (#624) Adds a basic `mlos_viz.plot(exp)` style API for simple visualizations of `ExperimentData` results relative to the experiment's objectives (building off of #628 and https://github.com/dabl/dabl/pull/335). Note: this PR currently omits unit tests for the new module due to the complexity of testing visualizations. We intend to add this in future PRs. There is however, a working example of its use here right now: https://github.com/Microsoft-CISL/sqlite-autotuning/pull/41 --------- Co-authored-by: Sergiy Matusevych <sergiy.matusevych@gmail.com>
2024-01-16 21:48:20 +03:00
rm -f build/pycodestyle.mlos_*.${CONDA_ENV_NAME}.build-stamp
Merged PR 1026: Split test and check rules and improve test coverage publishing in pipelines and docs - make test and check targets for mlos_core and mlos_bench independent for faster execution and more clear output for `make test` and `make check` iteration when only hacking on one module locally - ensures that the full set still runs in the pipeline though - add support for skipping code coverage locally by setting `SKIP_COVERAGE=true` to run tests faster - optionally copy the code coverage data into our static html site for publishing (plus nice badges) - also, rather than racing and publishing the test results and code coverage for all of the different sub jobs, we only run and upload coverage via `pytest-azurepipelines` for the main one we're interested in to make reporting easier - also includes which job the test came from in the results - some other general `Makefile` cleanup Future work: - [X] Split the single pipeline into two pipelines and make the Windows one optional (since it's usually the one that takes the longest and we care a little bit less about it now that we have decontainer support) !1040
2023-03-16 22:47:01 +03:00
rm -f build/pydocstyle.build-stamp
rm -f build/pydocstyle.${CONDA_ENV_NAME}.build-stamp
Introduce basic experiment visualization module mlos_viz via dabl (#624) Adds a basic `mlos_viz.plot(exp)` style API for simple visualizations of `ExperimentData` results relative to the experiment's objectives (building off of #628 and https://github.com/dabl/dabl/pull/335). Note: this PR currently omits unit tests for the new module due to the complexity of testing visualizations. We intend to add this in future PRs. There is however, a working example of its use here right now: https://github.com/Microsoft-CISL/sqlite-autotuning/pull/41 --------- Co-authored-by: Sergiy Matusevych <sergiy.matusevych@gmail.com>
2024-01-16 21:48:20 +03:00
rm -f build/pydocstyle.mlos_*.${CONDA_ENV_NAME}.build-stamp
Rework format lint checks for pre-commit (#904) # Pull Request ## Title Reworks build to use pre-commit for format and lint checks. --- ## Description As mentioned in #902 and #903, this refactors the build system to use [`pre-commit`](https://pre-commit.com) for the following reasons: - less complicated Makefile rules - pinned linter versions that can be checked with dependabot more explicitly for easier maintenance - See Also: #749 --- ## Type of Change - ✨ New feature - 🔄 Refactor - 📝 Documentation update - 🧪 Tests --- ## Testing Manually locally and usual CI tests. --- ## Additional Notes (optional) This builds off of #903 and is prep work to change the syntax of the codebase using `pyupgrade`. ---
2025-01-04 01:58:07 +03:00
rm -f build/check.${CONDA_ENV_NAME}.build-stamp
Merged PR 333: Initial MlosCore Implementation Adds basic `register` and `suggest` APIs for a couple of optimizer backends: `Skopt`, `Emukit`, `Random`. Includes example notebook for starters. Limited unit testing. Basic function documentation. Azure DevOps CI pipelines for `pylint` and `pytest` via the `mlos_core` `conda` `environment.yml`. Related work items: #274, #279
2022-03-14 22:37:52 +03:00
.PHONY: clean-test
clean-test:
Merged PR 903: Combination of additional build and doc fixups This PR was originally aimed at just fixing the intermittent documentation build errors. That issue ultimately stems from conflicts between conda and pip installs of two different versions of `mistune` dependency. Along the way I found a handful of other improvements for build, pipeline, and documentation generation things. - Make sure we do the nightly build regardless of whether code has changed - Don't build the docs in parallel by default (trying to avoid an intermittent build failure) - Add basic support for testing devcontainer manually outside for vscode - Stop doing parallel builds so it's easier to debug, and move doc build/publishing to the devcontainer - Split windows conda env off, stop using gpy from conda-forge, and just build it locally with clang, gcc, or msvc - Adjust the devcontainer build so that more dependencies are built upfront during the container build. This makes the process more cacheable so that we can just pull the upstream version and not have to build as much locally which could delay first launch. - Import part of a fix from elsewhere - Switch to docker-compose style devcontainer that automatically also starts an nginx container for viewing the generated docs - Update some docs and doc building - [X] pydocstyle fixups - [X] Add some log filtering of the `make doc` output for easier debugging (e.g. don't care about missing examples for now) - [X] doc link lint checks - DONE - [X] Split this out into multiple PRs - [X] SpaceAdapterFactory: !959 - [X] docstyle fixups: !960 - [ ] doc build fixups - [ ] devcontainer improvements (leaving the last two as this PR)
2023-02-27 08:42:22 +03:00
rm -f build/pytest.build-stamp
rm -f build/pytest.${CONDA_ENV_NAME}.build-stamp
Introduce basic experiment visualization module mlos_viz via dabl (#624) Adds a basic `mlos_viz.plot(exp)` style API for simple visualizations of `ExperimentData` results relative to the experiment's objectives (building off of #628 and https://github.com/dabl/dabl/pull/335). Note: this PR currently omits unit tests for the new module due to the complexity of testing visualizations. We intend to add this in future PRs. There is however, a working example of its use here right now: https://github.com/Microsoft-CISL/sqlite-autotuning/pull/41 --------- Co-authored-by: Sergiy Matusevych <sergiy.matusevych@gmail.com>
2024-01-16 21:48:20 +03:00
rm -f build/pytest.mlos_*.${CONDA_ENV_NAME}.build-stamp
rm -f build/pytest.mlos_*.${CONDA_ENV_NAME}.needs-build-stamp
Merged PR 561: Reorganize file layout to support separate whl outputs Related work items: #444
2022-08-04 23:26:40 +03:00
rm -rf .pytest_cache/
Merged PR 1026: Split test and check rules and improve test coverage publishing in pipelines and docs - make test and check targets for mlos_core and mlos_bench independent for faster execution and more clear output for `make test` and `make check` iteration when only hacking on one module locally - ensures that the full set still runs in the pipeline though - add support for skipping code coverage locally by setting `SKIP_COVERAGE=true` to run tests faster - optionally copy the code coverage data into our static html site for publishing (plus nice badges) - also, rather than racing and publishing the test results and code coverage for all of the different sub jobs, we only run and upload coverage via `pytest-azurepipelines` for the main one we're interested in to make reporting easier - also includes which job the test came from in the results - some other general `Makefile` cleanup Future work: - [X] Split the single pipeline into two pipelines and make the Windows one optional (since it's usually the one that takes the longest and we care a little bit less about it now that we have decontainer support) !1040
2023-03-16 22:47:01 +03:00
rm -f coverage.xml .coverage* build/.converage*
rm -rf htmlcov/
rm -rf junit/
rm -rf test-output.xml
Merged PR 333: Initial MlosCore Implementation Adds basic `register` and `suggest` APIs for a couple of optimizer backends: `Skopt`, `Emukit`, `Random`. Includes example notebook for starters. Limited unit testing. Basic function documentation. Azure DevOps CI pipelines for `pylint` and `pytest` via the `mlos_core` `conda` `environment.yml`. Related work items: #274, #279
2022-03-14 22:37:52 +03:00
Staged Makefile Improvements (#762) - Introduce a new `make format` target that currently only calls `licenseheaders` but will call `isort` and `black` in the future - Introduce new variables to help make dependency tracking easier - Simplify `pytest` rules and allow running only a single module at a time. This PR is a precursor to pyproject.toml related build changes as well. --------- Co-authored-by: Sergiy Matusevych <sergiym@microsoft.com>
2024-07-01 18:57:17 +03:00
.PHONY: clean-dist
clean-dist:
rm -rf dist
Merged PR 600: Fixup wheel generation
2022-09-01 00:49:31 +03:00
rm -rf mlos_core/build mlos_core/dist
rm -rf mlos_bench/build mlos_bench/dist
Introduce basic experiment visualization module mlos_viz via dabl (#624) Adds a basic `mlos_viz.plot(exp)` style API for simple visualizations of `ExperimentData` results relative to the experiment's objectives (building off of #628 and https://github.com/dabl/dabl/pull/335). Note: this PR currently omits unit tests for the new module due to the complexity of testing visualizations. We intend to add this in future PRs. There is however, a working example of its use here right now: https://github.com/Microsoft-CISL/sqlite-autotuning/pull/41 --------- Co-authored-by: Sergiy Matusevych <sergiy.matusevych@gmail.com>
2024-01-16 21:48:20 +03:00
rm -rf mlos_viz/build mlos_viz/dist
Merged PR 333: Initial MlosCore Implementation Adds basic `register` and `suggest` APIs for a couple of optimizer backends: `Skopt`, `Emukit`, `Random`. Includes example notebook for starters. Limited unit testing. Basic function documentation. Azure DevOps CI pipelines for `pylint` and `pytest` via the `mlos_core` `conda` `environment.yml`. Related work items: #274, #279
2022-03-14 22:37:52 +03:00
.PHONY: clean
Staged Makefile Improvements (#762) - Introduce a new `make format` target that currently only calls `licenseheaders` but will call `isort` and `black` in the future - Introduce new variables to help make dependency tracking easier - Simplify `pytest` rules and allow running only a single module at a time. This PR is a precursor to pyproject.toml related build changes as well. --------- Co-authored-by: Sergiy Matusevych <sergiym@microsoft.com>
2024-07-01 18:57:17 +03:00
clean: clean-format clean-check clean-test clean-dist clean-doc clean-doc-env clean-dist-test
Introduce pyproject.toml build changes (#764) This PR builds off of #762 and #763 These are in part 1. followup fixups for #746 (e.g., to allow setuptools-scm to be pulled in at build time as a build dependency only when conda an pip have mismatched version issues), and 2. Modernization improvements to allow us to make better use of other tools (e.g., `black` that only accept `pyproject.toml` files as their configuration files). To do so, we move some configs from `setup.py` to `pyproject.toml` for each module. However, to retain the ability to rewrite URLs in published README.md files on PyPi as well as consistent version dependencies across modules without the need to manually specify version numbers (e.g., using `setuptools-scm`) we mark a few dependencies as dynamic and leave our existing logic inside the `setup.py` file. Finally, we reorganize the `version.py` file to be inside the module and fix a few previous omissions for `mlos_viz`.
2024-07-03 01:31:31 +03:00
rm -f build/unshallow.build-stamp
Merged PR 903: Combination of additional build and doc fixups This PR was originally aimed at just fixing the intermittent documentation build errors. That issue ultimately stems from conflicts between conda and pip installs of two different versions of `mistune` dependency. Along the way I found a handful of other improvements for build, pipeline, and documentation generation things. - Make sure we do the nightly build regardless of whether code has changed - Don't build the docs in parallel by default (trying to avoid an intermittent build failure) - Add basic support for testing devcontainer manually outside for vscode - Stop doing parallel builds so it's easier to debug, and move doc build/publishing to the devcontainer - Split windows conda env off, stop using gpy from conda-forge, and just build it locally with clang, gcc, or msvc - Adjust the devcontainer build so that more dependencies are built upfront during the container build. This makes the process more cacheable so that we can just pull the upstream version and not have to build as much locally which could delay first launch. - Import part of a fix from elsewhere - Switch to docker-compose style devcontainer that automatically also starts an nginx container for viewing the generated docs - Update some docs and doc building - [X] pydocstyle fixups - [X] Add some log filtering of the `make doc` output for easier debugging (e.g. don't care about missing examples for now) - [X] doc link lint checks - DONE - [X] Split this out into multiple PRs - [X] SpaceAdapterFactory: !959 - [X] docstyle fixups: !960 - [ ] doc build fixups - [ ] devcontainer improvements (leaving the last two as this PR)
2023-02-27 08:42:22 +03:00
rm -f .*.build-stamp
rm -f build/conda-env.build-stamp build/conda-env.*.build-stamp
Merged PR 380: Add conda env caching for build pipeline - Add conda env caching for build pipeline speed ups. - Cache will use items from the last hour (within the same day), but only if the `environment.yml` and `setup.yml` file haven't changed. - Adds testing for multiple Python versions. - Mostly as a proof of concept. Right now just 3.8 and 3.9. 3.6 (the current default in Synapse) is broken and disabled. - We can disable this and just use the `DefaultPython` version again for now as well. - Also enabled parallel builds as a part of this change. Related work items: #322
2022-03-17 00:32:53 +03:00
rm -rf mlos_core.egg-info
Merged PR 561: Reorganize file layout to support separate whl outputs Related work items: #444
2022-08-04 23:26:40 +03:00
rm -rf mlos_core/mlos_core.egg-info
Merged PR 560: Merging initial os_autotune_main work to main We will do our OS autotuning related work on mlos-bench there. Main prior to this has been tagged as v0.0.3 for servicing Spark autotune needs if need be. Plan is to reorganize the source tree after this so we can generate separate `mlos-core` and `mlos-bench` packages.
2022-08-04 19:11:35 +03:00
rm -rf mlos_bench.egg-info
Merged PR 561: Reorganize file layout to support separate whl outputs Related work items: #444
2022-08-04 23:26:40 +03:00
rm -rf mlos_bench/mlos_bench.egg-info
Introduce basic experiment visualization module mlos_viz via dabl (#624) Adds a basic `mlos_viz.plot(exp)` style API for simple visualizations of `ExperimentData` results relative to the experiment's objectives (building off of #628 and https://github.com/dabl/dabl/pull/335). Note: this PR currently omits unit tests for the new module due to the complexity of testing visualizations. We intend to add this in future PRs. There is however, a working example of its use here right now: https://github.com/Microsoft-CISL/sqlite-autotuning/pull/41 --------- Co-authored-by: Sergiy Matusevych <sergiy.matusevych@gmail.com>
2024-01-16 21:48:20 +03:00
rm -rf mlos_viz.egg-info
rm -rf mlos_viz/mlos_viz.egg-info
Merged PR 561: Reorganize file layout to support separate whl outputs Related work items: #444
2022-08-04 23:26:40 +03:00
rm -rf __pycache__
find . -type d -name __pycache__ -print0 | xargs -t -r -0 rm -rf
find . -type f -name '*.pyc' -print0 | xargs -t -r -0 rm -f
Merged PR 903: Combination of additional build and doc fixups This PR was originally aimed at just fixing the intermittent documentation build errors. That issue ultimately stems from conflicts between conda and pip installs of two different versions of `mistune` dependency. Along the way I found a handful of other improvements for build, pipeline, and documentation generation things. - Make sure we do the nightly build regardless of whether code has changed - Don't build the docs in parallel by default (trying to avoid an intermittent build failure) - Add basic support for testing devcontainer manually outside for vscode - Stop doing parallel builds so it's easier to debug, and move doc build/publishing to the devcontainer - Split windows conda env off, stop using gpy from conda-forge, and just build it locally with clang, gcc, or msvc - Adjust the devcontainer build so that more dependencies are built upfront during the container build. This makes the process more cacheable so that we can just pull the upstream version and not have to build as much locally which could delay first launch. - Import part of a fix from elsewhere - Switch to docker-compose style devcontainer that automatically also starts an nginx container for viewing the generated docs - Update some docs and doc building - [X] pydocstyle fixups - [X] Add some log filtering of the `make doc` output for easier debugging (e.g. don't care about missing examples for now) - [X] doc link lint checks - DONE - [X] Split this out into multiple PRs - [X] SpaceAdapterFactory: !959 - [X] docstyle fixups: !960 - [ ] doc build fixups - [ ] devcontainer improvements (leaving the last two as this PR)
2023-02-27 08:42:22 +03:00
.PHONY: devcontainer
devcontainer:
./.devcontainer/build/build-devcontainer.sh
@echo
Merged PR 990: Try and use an inline container cache to speed up CI pipelines and reuse remotely built container locally
2023-03-09 05:58:39 +03:00
@echo "Run ./.devcontainer/scripts/run-devcontainer.sh to start the newly built devcontainer."