As outlined in Issue 9104 the python subprocess return code can be less than 0.
The previous version only captures errors of the subprocess itself, not the negative error codes, as cause by host SIGKILL, SIGHUP, ... see https://docs.python.org/3/library/subprocess.html#subprocess.
CompletedProcess.returncode
We now treat all non-zero returncodes as a script failure.
The previous method of generating this list had two "problems"/niggles
that this PR solves, when running with VERBOSE=true
- Firstly, LOCAL_MOUNTS was set at the top level, so running with
`set -x` produced 30 extra lines of output.
- Because of the `while read` used, it created 4 or 5 lines _per_ mount,
resulting in a lot verbose output.
Nothing I've changed here is "critical", it's just making it a bit
easier to see with the debug output what is going on, by running fewer
commands.
I have also expanded the BATS test a little bit to check each pair (`-v`
and its following option)
Breeze had --push-images switch to also push images to repo
but it was often needed to build and push images separately.
We have now a possibility to push an already built image with
separate push-image command instead and also you can choose
to push to cache registry in GitHub rather than to DockerHub
with --registry-cache switch.
scripts/ci/*.sh uses MY_DIR to mean scripts/ci, but in `breeze` MY_DIR
is the same as AIRFLOW_SOURCES. When jumping back-and-forth between
ci/_utils.sh, breeze, and ci/ci_*.sh it can be confusing to keep track
of what is what.
This changes `breeze` to use `AIRFLOW_SOURCES` to referrer to the top
level folder instead -- that means I don't have to keep as much context
in my head
* Add OpenAPI spec (#7549)
* Fix typo in name of pre-commit hook
* Chaange type for DAGID, DAGRunID, TaskID
* Fix typo in summary - POST /pools
* Fix typo in description - FileToken parameter
* Fix typo - singular/plural form - variables
* Make EventLog endpoints read-only
* Use ExcutionDate in DagRuns endpoints
* Use custom action to control task instances
* Typo in DELETE Task instance
* Remove unused schema - DagStructureCollection
* Fix typo - singular/plural form - import errors
* Add endpoint - POST /dagRuns
* Remove job_id
We do not have endpoints to download jobs, because this is an implementation detail, so this field has big no value.
* Add filters to GET /taskInstances
* Fix typo - upadtePool => updatePool
* Rename "Create a DAG Run" to "Trigger a DAG Run"
* Use Pool name as a parameter
* Add filter to GET /dagRuns
* Remove invalid note ion start_date field
* Uss POST instead of PATCH for custom action
* Remove DELETE /taskInstances endpooint
* Rename Xcom Value to xcom Entry
* Fix typo in XCCOM Entry endpoint
* Change operationID: patchConnection => updaateConnection
* Make execution_date optionall in DAGRun
This field can be filled with the current date when creating the DAG Run
* Unify connection ID
* Use URL with HTTPS and without www.
* Fix typo - at database => in database
* Fix typo = Raange -> Raange
* Fix typo - the specific DAG => a DAG
* Fix typo - getXComVEntry => getXComVEntry
* Unify collection names - xcomEntries
* Move TaskInstance resource under DagRun resource
* Fix typo - change tag - TaskInstance => TaskInstance
Co-authored-by: Ash Berlin-Taylor <ash_github@firemirror.com>
* Use path paramaters for /variables/lookup/ endpoint
* Use consistent names for IDs
* Use new style for filter parameters
* Remove unused path parameter
* Use ~ as a wildcard charaacter
* Add batch endpoints for TaskInstance and DagRuns
* Fix typo - response in trigger dag endpoint
* Fix typo - Qqueue => Queue
* Set dry_run = True in ClearTaskInstance
* Mark all fieldss (expcet state) of DagRun as read-only
* Use __type as a discriminator
* Fix typo - "The Import Error ID." => "The Event Log ID."
* Fix typo - Self referential in EventLogCollection
* Rename fieldss - dttm => when
* remove fields - pool_id
* Fix typo - change request body in PATCH /pools/{pool_name}
* Use DAG Run ID as a primary identifier
* Fix typo - Change type of query to string
* Unify fields names in collections
* Use variable key as a primary id
* Move collection to /variables
* Mark passord as a write only
* Fix typo - updaateConnection => updateConnection
* Change is_paused/is_subdag to boolean
* Fix typo - clearTaskInstaance => clearTaskInstance
* Fix typo - DAAG => DAG
* Fix typo - many => multiple
* Fix typo - missing "a"
* Fix typo - variable by id => variable by key
* Fix typo - updateXComEntries => updateXComEntry
* Fix typo - missing "a"
* Use dag_run_id as a primary ID
* Fix typo - objectss => objects, DAG IDS => DAG IDs
* Allows create DAG Run with custom conf/execution_date/dag_run_id
* Add new trigger rule, fix typo in dag run state
* Add request body to POST/PATCH /dags/{dag_id}
* Rename collection fields - dag_model => dags
* Fix typo - /clearTaskInstanaces -> /clearTaskInstances
* Improve wording - wildcard
* Returns owners as a array
* Return only references in clear task instances
* Remove support for application/x-www-form-urlencoded
* fixup! Use __type as a discriminator
* Add file_token fields
* Move description of variable collections
* Return SUB DAG in DAG structure
* Fix typo - sucess => sucess, Apache Foundation => Apache Software Foundation, Airfow => Apache Airflow
* Improve description of get logs endpoint
* Fix typo - Get all XCom entry => Get all XCom entries
* Add crossreference between /dags/{dag_id}/structure and /dags/{dag_id}
* Remove all form-urllencoded request bodies
* Rename parameter - NoChunking => FullContent
* Improve description of batch endpoints
* Remove request body for GET endpoint
* Use allOf insteaad of oneOf
* Rename key => xcom_key
* Use lowercase letters in query string parameter - Queue -> queue
* Change type of conf to object
* Change allOf into oneOf for ScheduleInterval
Co-authored-by: Ash Berlin-Taylor <ash_github@firemirror.com>
* Add SQL Branch Operator
SQL Branch Operator allow user to execute a SQL query in any supported backend to decide which
branch to follow. The SQL branch operator expect query to return True/False (Boolean) or
0/1 (Integer) or true/y/yes/1/on/false/n/no/0/off (String).
* add description to subcommands and move them to respective subcommands
* move some command help to description
* add default description
* improve text
* move some help text to description
* improve code
The "next_execution" cli sub-command now accepts an optional number of
executions to be returned. This is particularly useful for checking
non-regular schedule intervals, such as those created by some cron
expressions.
Co-authored-by: Kamil Breguła <mik-laj@users.noreply.github.com>
* added documentation for dataproc
* added more update information for updateMask
* Added link to information about cluster config api request
* Apply naming convention
* Set all dedents from 4 to 0
* Adjust dedent to 4, for operators
* removed dataproc guide from test_missing_guides
As part of the scheduler HA work we are going to want to separate the
parsing from the scheduling, so this changes the tests to ensure that
the important methods of DagFileProcessor can do everything the need to
when given a SerializedDAG, not just a DAG. i.e. that we have correctly
serialized all the necessary fields.
* Small style changes in BaseJob
* Small code improvements in SchedulerJob
* fixup! Small code improvements in SchedulerJob
* fixup! fixup! Small code improvements in SchedulerJob
* Improve test for the next_execution cli command
It still tests the same functionality, but it is now more efficient and
readable. The changes are:
- The DB is cleaned only once instead of eight times
- Use a fixed datetime instead of timezone.utcnow (deterministic)
- Use redirect_stdout instead of subprocess
- Clean up pylint warning
- Test None output once instead of four times
* Address PR comments
Issues addressed:
- Use create_session
- Use DagRunType.MANUAL.value
- Remove DagRuns created by the test
* Remove test from quarantine
Co-authored-by: Tomek Urbaszek <turbaszek@gmail.com>
Co-authored-by: Tomek Urbaszek <turbaszek@gmail.com>
* Added SlackAPIFileOperator
* Added usage example in docstring
* Added tests for SlackAPIFileOperator, fixed extra line and added missing type hinting in operators/slack.py
* Fixed import order for isort
* Refactor conn_id for slack_conn_id
Co-authored-by: Kaxil Naik <kaxilnaik@gmail.com>
* Remove # from channel name for default channel name
Co-authored-by: Kaxil Naik <kaxilnaik@gmail.com>
Co-authored-by: Mikaël Ducharme <mikaelducharme@effenco.com>
Co-authored-by: Kaxil Naik <kaxilnaik@gmail.com>
* Use production image for k8s tests
The CI image has become too large to load into KinD,
it also only really makes sense to use the production image for
integration tests
* nit
Co-authored-by: Daniel Imberman <daniel@astronomer.io>
Kamil Breguła
James Timmins
Tomek Urbaszek
Kaxil Naik
Sebastian Werner
Ash Berlin-Taylor
Jarek Potiuk
mgorsk1
samuelkhtu
Ephraim Anierobi
Damian
Mauricio De Diana
Joppe Vos
Ace Haidrey
Bas Harenslak
S S Rohit
Mikaël Ducharme
JavierLopezT
Yingbo Wang
Daniel Imberman
QP Hou
Daniel Huang
Andrej Švec