* Define dev requirements in `requirements-dev.in`.
* Add `pip-tools` as a dev requirement.
* Remove `pip-compile` as a prerequisite for `pip-install-local`.
To avoid the bootstrapping issue in a fresh Python environment where `pip-compile` isn't installed yet.
* Update `.PHONY` config in `Makefile` to include all non-file targets.
* Update `help` target in `Makefile` to include all available commands.
* Replace tabs in `Makefile` strings with spaces.
* Update CI instructions for fixing lint errors.
* Upgrade Airflow from 2.7.3 to 2.8.2.
* Update CircleCI Docker orb from 2.2.0 to 2.5.0.
* Update Ruff config for version 0.2.
* fix dockerfile URL scheme to use https for gcloud CLI
* Restore `google-auth` extra, with its built-in dependency on `authlib`.
* Add missing Airflow state colors for "restarting", "deferred", and "removed".
* Explain the difference between our configured Airflow state colors and the defaults.
---------
Co-authored-by: mikaeld <mducharme@mozilla.com>
* CI runtime decreased by 70-80%
* docker-compose setup time decreased significantly. **Local use: takes a few seconds as opposed to a few minutes before**. Airflow Variables and Connections are loaded via `import` CLI command; replaces `bin/run` script using Airflow CLI.
* Improved dev secrets security by dynamically generating a Fernet key in makefile
* `.env` file is generated from `make up`, contains UID and Fernet key. `.env` is automatically loaded into environment variables by `docker-compose`
* Retire shell scripts to use builtin features in our stack
* `bin/run` replaced by docker-compose and CI
* `bin/test-dag-tags` replaced by pytest unit tests
* `bin/test-parse` replaced by pytest unit tests
* Revert "update airflow config for 2.3.3"
This reverts commit d19cc711aa.
* Revert "fix deprecation warnings, clean up and update for 2.3.3"
This reverts commit e80472ab9a.
* Revert "update requirements, introduce constraints file and clean up for 2.3.3"
This reverts commit 8e60dba783.
* script to validate dag tags and step to circle ci
* trying out dag tagg validation through parsing
* added missing tag so that tag check does not fail
* Using SQL approach for validation, added extra logging and clean up
* added check to make sure all DAGs have tags
* fixed 3 DAGs missing tags
* implemented suggestions by @haroldwoo in #1446
Co-authored-by: = <=>
* Airflow tags added + tests + pytest to requirements
* added pytest execution to test stage in circleci pipeline
* moved tests out of dags/
Co-authored-by: = <=>
* Upgrade to 1.10.10 and pip-compile using python3.8
* pip-compile with python3.7
* Update Dockerfile from 2.7-slim to 3.7-slim
* Remove deprecated option from config
* Update relative imports in backported operators
* Replace urlparse to new location
* Replace iteritems with items
* Add instructions for pip-compile and cut out old material
* Update unittests for plugins
* Update README.md
Co-authored-by: Daniel Thorn <dthorn@mozilla.com>
* Update README.md
Co-authored-by: Daniel Thorn <dthorn@mozilla.com>
* Compile requirements using explicit source
* Add CI for checking requirements are generated correctly
* Leave off requirements.in from pip-compile call
* Recompile requirements.txt
* Pin kombu to 4.6.3
Co-authored-by: Daniel Thorn <dthorn@mozilla.com>
* Add testing script to parse dags in CI
* [testing] Add a broken dag to test CI changes
* [testing] Remove testing file for CI
* Update .circleci/config.yml
Co-Authored-By: Sunah Suh <github@sunahsuh.com>
* Address review comments
Use docker caching layer and a more recent ubuntu image for the machine
executor
* Add initial statuspage operator to plugins
* Make DatasetStatusOperator operational
* [example] DatasetStatusOperator usage
* Format code with black
* Add tests for the DatasetStatusOperator
* Add tests for DatasetStatusHook
* Do not use relative importing with plugins
* Add tests for dags.utils.status
* Add register_status to utils
* Add tox for running pytests
* Add tox to CircleCI
* Separate dataset and status page clients
* Comment on fixing sys.module path
* Raise on non-200 status
* Add comments to the airflow operator
* Add a default on_success=False option to register_status
Mikaël Ducharme
Anna Scholtz
Sean Rose
kik-kik
Harold Woo
Wesley Dawson
Anthony Miyaguchi
Anthony Miyaguchi