github
/
incubator-airflow
зеркало из https://github.com/github/incubator-airflow.git
1
0
Форкнуть 0
Код Задачи Пакеты Проекты Релизы Вики Активность
incubator-airflow/breeze

3085 строки
111 KiB
Plaintext
Исходник Обычный вид История

[AIRFLOW-3611] Simplified development environment (#4932)
2019-08-27 21:39:36 +03:00
#!/usr/bin/env bash
[AIRFLOW-5750] Licence check is done also for non-executable .sh (#6425)
2019-10-25 10:54:36 +03:00
[AIRFLOW-3611] Simplified development environment (#4932)
2019-08-27 21:39:36 +03:00
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
set -euo pipefail
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
AIRFLOW_SOURCES="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
export AIRFLOW_SOURCES
readonly AIRFLOW_SOURCES
[AIRFLOW-3611] Simplified development environment (#4932)
2019-08-27 21:39:36 +03:00
[AIRFLOW-6820] split breeze into functions (#7433)
2020-02-19 03:58:53 +03:00
# Bash arrays need to be defined outside of functions unfortunately :(
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
# Because on Mac OS Bash 3.4 defining arrays inside functions does not work
[AIRFLOW-6820] split breeze into functions (#7433)
2020-02-19 03:58:53 +03:00
# Array with extra options for Docker compose
declare -a EXTRA_DC_OPTIONS
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
export EXTRA_DC_OPTIONS
[AIRFLOW-6820] split breeze into functions (#7433)
2020-02-19 03:58:53 +03:00
# Array with selected integrations
declare -a INTEGRATIONS
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
export INTEGRATIONS
[AIRFLOW-6820] split breeze into functions (#7433)
2020-02-19 03:58:53 +03:00
# This is where remaining args are passed
declare -a REMAINING_ARGS
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
export REMAINING_ARGS
[AIRFLOW-6841] Fixed unbounded variable on Mac (#7465)
2020-02-19 19:35:15 +03:00
# This is where static check options are defined
declare -a EXTRA_STATIC_CHECK_OPTIONS
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
export EXTRA_STATIC_CHECK_OPTIONS
[AIRFLOW-3611] Simplified development environment (#4932)
2019-08-27 21:39:36 +03:00
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
#######################################################################################################
# Sets up all the default variables for Breeze. They are needed by all other functions
# All those variables are exported. They are not set to read-only because those
# defaults can be modified later on when command line arguments are parsed
# and variables stored in .build directory (stored in the previous run) are read
#
# Used globals:
# FORCE_SCREEN_WIDTH
#
# Modified globals (constants or candidates for constants after we override them via appropriate flags):
#
# BREEZE
# SUPPRESS_CHEATSHEET_FILE
# SUPPRESS_ASCIIART_FILE
# MAX_SCREEN_WIDTH
# SCREEN_WIDTH
# MOUNT_LOCAL_SOURCES
# MOUNT_FILES
# FORCE_PULL_IMAGES
# ENABLE_KIND_CLUSTER
# FORWARD_CREDENTIALS
# DB_RESET
Flag --start-airflow for breeze (#10837)
2020-09-12 00:26:56 +03:00
# START_AIRFLOW
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
# INSTALL_AIRFLOW_VERSION
# INSTALL_AIRFLOW_REFERENCE
# FORCE_BUILD_IMAGES
# PRODUCTION_IMAGE
# PYTHON_MAJOR_MINOR_VERSION
#
# Global variables:
#
# command_to_run
# second_command_to_run
# docker_compose_command
#
# Also it sets the variables and globals set by common initialization functions from
# scripts/ci/libraries/_initialization.sh and breeze-complete script (which sets-up auto-complete).
#
#######################################################################################################
Add packages to function names in bash (#10670) (#10696) Inspired by the Google Shell Guide where they mentioned separating package names with :: I realized that this was one of the missing pieces in the bash scripts of ours. While we already had packages (in libraries folders) it's been difficult to realise which function is where. With introducing packages - equal to the library file name we are *almost* at a level of a structured language - and it's easier to find the functions if you are looking for them. Way easier in fact. Part of #10576 (cherry picked from commit cc551ba793344800d2d396c13d7fd0c8eed97352) (cherry picked from commit 2bba276f0f06a5981bdd7e4f0e7e5ca2fe84f063)
2020-09-02 22:58:37 +03:00
function breeze::setup_default_breeze_constants() {
Add comments to breeze scripts (#8797)
2020-05-10 20:55:40 +03:00
# Indicates that we are inside Breeze environment
[AIRFLOW-6820] split breeze into functions (#7433)
2020-02-19 03:58:53 +03:00
export BREEZE=true
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
readonly BREEZE
[AIRFLOW-3611] Simplified development environment (#4932)
2019-08-27 21:39:36 +03:00
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
# If those files are present, the ASCII-art/cheat-sheet are suppressed
SUPPRESS_CHEATSHEET_FILE="${AIRFLOW_SOURCES}/.build/.suppress_cheatsheet"
readonly SUPPRESS_CHEATSHEET_FILE
Revert recent breeze changes (#10651 & #10670) (#10694) * Revert "Add packages to function names in bash (#10670)" This reverts commit cc551ba793344800d2d396c13d7fd0c8eed97352. * Revert "Implement Google Shell Conventions for breeze script … (#10651)" This reverts commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2.
2020-09-02 19:27:36 +03:00
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
SUPPRESS_ASCIIART_FILE="${AIRFLOW_SOURCES}/.build/.suppress_asciiart"
readonly SUPPRESS_ASCIIART_FILE
Revert recent breeze changes (#10651 & #10670) (#10694) * Revert "Add packages to function names in bash (#10670)" This reverts commit cc551ba793344800d2d396c13d7fd0c8eed97352. * Revert "Implement Google Shell Conventions for breeze script … (#10651)" This reverts commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2.
2020-09-02 19:27:36 +03:00
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
# Maximum screen indented_screen_width to print the lines spanning the whole terminal indented_screen_width
export MAX_SCREEN_WIDTH=100
readonly MAX_SCREEN_WIDTH
[AIRFLOW-3611] Simplified development environment (#4932)
2019-08-27 21:39:36 +03:00
Add comments to breeze scripts (#8797)
2020-05-10 20:55:40 +03:00
# By default we mount local Airflow sources
Update github flags for Breeze (#10384) Part of #10368
2020-08-19 01:53:04 +03:00
export MOUNT_LOCAL_SOURCES="true"
[AIRFLOW-3611] Simplified development environment (#4932)
2019-08-27 21:39:36 +03:00
Replace file.io with artifacts (#10137)
2020-08-04 02:26:59 +03:00
# By default we mount files folder
Update github flags for Breeze (#10384) Part of #10368
2020-08-19 01:53:04 +03:00
export MOUNT_FILES="true"
Replace file.io with artifacts (#10137)
2020-08-04 02:26:59 +03:00
[AIRFLOW-6820] split breeze into functions (#7433)
2020-02-19 03:58:53 +03:00
# By default we only pull images if we do not have them locally.
Add comments to breeze scripts (#8797)
2020-05-10 20:55:40 +03:00
# This can be overridden by '--force-pull-images' flag
[AIRFLOW-6820] split breeze into functions (#7433)
2020-02-19 03:58:53 +03:00
export FORCE_PULL_IMAGES="false"
[AIRFLOW-3611] Simplified development environment (#4932)
2019-08-27 21:39:36 +03:00
Kubernetes Cluster is started on host not in the container (#8265) Tests requiring Kubernetes Cluster are now moved out of the regular CI tests and moved to "kubernetes_tests" folder so that they can be run entirely on host without having the CI image built at all. They use production image to run the tests on KinD cluster and we add tooling to start/stop/deploy the application to the KinD cluster automatically - for both CI testing and local development. This is a pre-requisite to convert the tests to convert the tests to use the official Helm Chart and Docker images or Apache Airflow. It closes #8782
2020-06-03 21:58:38 +03:00
# Do not enable Kind Kubernetes cluster by default
export ENABLE_KIND_CLUSTER="false"
Add comments to breeze scripts (#8797)
2020-05-10 20:55:40 +03:00
# Forward common host credentials to docker (gcloud, aws etc.).
[AIRFLOW-6763] Make systems tests ready for backport tests (#7389) We will run system test on back-ported operators for 1.10* series of airflow and for that we need to have support for running system tests using pytest's markers and reading environment variables passed from HOST machine (to pass credentials).  This is the first step to automate system tests execution.
2020-02-21 20:25:32 +03:00
export FORWARD_CREDENTIALS="false"
Add comments to breeze scripts (#8797)
2020-05-10 20:55:40 +03:00
# If set to true, the database will be reset at entry. Works for Postgres and MySQL
[AIRFLOW-7054] Breeze has an option now to reset db at entry (#7710)
2020-03-13 18:08:16 +03:00
export DB_RESET="false"
Revert "Introducing flags to skip example dags and default connections (#11099)" (#11110) This reverts commit 0edc3dd57953da5c66a4b45d49c1426cc0295f9e.
2020-09-23 21:47:43 +03:00
# If set to true, the database will be initialized, a user created and webserver and scheduler started
export START_AIRFLOW="false"
New Breeze command start-airflow, it replaces the previous flag (#11157)
2020-09-27 19:31:50 +03:00
# If set to true, the test connections will be created
export LOAD_DEFAULT_CONNECTIONS="false"
# If set to true, the sample dags will be created
export LOAD_EXAMPLES="false"
Help for breeze commands contain relevant flags. (#8261) * Help for breeze commands contain relevant flags. * fixup! Help for breeze commands contain relevant flags. * fixup! fixup! Help for breeze commands contain relevant flags.
2020-04-13 09:51:33 +03:00
# If it set is set to specified version, then the source version of Airflow
# is removed and the specified version of Airflow is installed from PyPi
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
export INSTALL_AIRFLOW_VERSION=""
[AIRFLOW-6763] Make systems tests ready for backport tests (#7389) We will run system test on back-ported operators for 1.10* series of airflow and for that we need to have support for running system tests using pytest's markers and reading environment variables passed from HOST machine (to pass credentials).  This is the first step to automate system tests execution.
2020-02-21 20:25:32 +03:00
Help for breeze commands contain relevant flags. (#8261) * Help for breeze commands contain relevant flags. * fixup! Help for breeze commands contain relevant flags. * fixup! fixup! Help for breeze commands contain relevant flags.
2020-04-13 09:51:33 +03:00
# If it is set to specified reference (tag/branch), then the source version
# of Airflow is removed and the specified version of Airflow is installed from GitHub
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
export INSTALL_AIRFLOW_REFERENCE=""
Add Production Docker image support (#7832)
2020-04-02 20:52:11 +03:00
Add comments to breeze scripts (#8797)
2020-05-10 20:55:40 +03:00
# Determines whether to force build without checking if it is needed
# Can be overridden by '--force-build-images' flag.
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
export FORCE_BUILD_IMAGES="false"
Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals.
2020-08-31 14:24:53 +03:00
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
# load all the common functions here - those are the functions that are shared between Breeze
# and CI scripts. The CI scripts do not use Breeze as driving script - they read all configuration
# from the environment variables. That's why we keep all the common initialization in those libs
# shellcheck source=scripts/ci/libraries/_all_libs.sh
. "${AIRFLOW_SOURCES}/scripts/ci/libraries/_all_libs.sh"
# When we generate documentation for README files, we want to force the indented_screen_width of terminal so that
# No matter who is running the documentation generation gets the same output
if [[ ${FORCE_SCREEN_WIDTH:="false"} != "true" ]]; then
# Sets indented_screen_width of the screen from terminal
SCREEN_WIDTH="$(tput cols)"
if [[ -z ${SCREEN_WIDTH=} ]]; then
SCREEN_WIDTH=${MAX_SCREEN_WIDTH}
fi
if ((SCREEN_WIDTH > MAX_SCREEN_WIDTH)); then
SCREEN_WIDTH=${MAX_SCREEN_WIDTH}
fi
else
SCREEN_WIDTH=${MAX_SCREEN_WIDTH}
fi
export SCREEN_WIDTH
readonly SCREEN_WIDTH
# Update short and long options in the breeze-complete script
# This way autocomplete will work automatically with all options available
# shellcheck source=breeze-complete
. "${AIRFLOW_SOURCES}/breeze-complete"
# Default command to run - entering breeze environment
command_to_run="enter_breeze"
# In some cases we also want to run two commands in a row (for example when we restart the environment)
second_command_to_run=""
# Determines if help should be run (set to true by --help flag)
run_help="false"
# Holds docker compose command if the `docker-compose` command is used.
docker_compose_command=""
[AIRFLOW-6820] split breeze into functions (#7433)
2020-02-19 03:58:53 +03:00
}
[AIRFLOW-3611] Simplified development environment (#4932)
2019-08-27 21:39:36 +03:00
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
#######################################################################################################
#
Add comments to breeze scripts (#8797)
2020-05-10 20:55:40 +03:00
# Initializes development-friendly virtualenv if you are already in such env. It installs all the necessary
# packages from PyPI and it case of problems it provides useful hints on what prerequisites should be
Release candidate 2 for backport packages 2020.05.20 (#8898) Release candidate 2 for backport packages 2020.05.20
2020-05-19 15:17:22 +03:00
# installed. It also removes and resets the existing AIRFLOW_HOME installation to make sure that you
Add comments to breeze scripts (#8797)
2020-05-10 20:55:40 +03:00
# have it synchronized with the version of airflow installed. It resets the airflow's sqlite database to
# a clean state. You can use this function if your virtualenv is broken, to clean it up
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
#
# Used globals:
# PYTHON_MAJOR_MINOR_VERSION
# AIRFLOW_HOME_DIR
# AIRFLOW_SOURCES
# DEFAULT_CONSTRAINTS_BRANCH
# OSTYPE
#
#######################################################################################################
Add packages to function names in bash (#10670) (#10696) Inspired by the Google Shell Guide where they mentioned separating package names with :: I realized that this was one of the missing pieces in the bash scripts of ours. While we already had packages (in libraries folders) it's been difficult to realise which function is where. With introducing packages - equal to the library file name we are *almost* at a level of a structured language - and it's easier to find the functions if you are looking for them. Way easier in fact. Part of #10576 (cherry picked from commit cc551ba793344800d2d396c13d7fd0c8eed97352) (cherry picked from commit 2bba276f0f06a5981bdd7e4f0e7e5ca2fe84f063)
2020-09-02 22:58:37 +03:00
function breeze::initialize_virtualenv() {
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
# Check if we are inside virtualenv
set +e
echo -e "import sys\nif not hasattr(sys,'base_prefix'):\n sys.exit(1)" |
python"${PYTHON_MAJOR_MINOR_VERSION}"
local res=$?
set -e
if [[ ${res} != "0" ]]; then
[AIRFLOW-6820] split breeze into functions (#7433)
2020-02-19 03:58:53 +03:00
echo >&2
echo >&2 "ERROR: Initializing local virtualenv only works when you have virtualenv activated"
echo >&2
Help for breeze commands contain relevant flags. (#8261) * Help for breeze commands contain relevant flags. * fixup! Help for breeze commands contain relevant flags. * fixup! fixup! Help for breeze commands contain relevant flags.
2020-04-13 09:51:33 +03:00
echo >&2 "Please enter your local virtualenv before (for example using 'pyenv activate' or 'workon') "
[AIRFLOW-6820] split breeze into functions (#7433)
2020-02-19 03:58:53 +03:00
echo >&2
exit 1
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
else
[AIRFLOW-6820] split breeze into functions (#7433)
2020-02-19 03:58:53 +03:00
echo
echo "Initializing the virtualenv: $(command -v python)!"
echo
echo "This will wipe out ${AIRFLOW_HOME_DIR} and reset all the databases!"
echo
Don't reuse MY_DIR in breeze to mean different folder from ci/_utils.sh (#9098) scripts/ci/*.sh uses MY_DIR to mean scripts/ci, but in `breeze` MY_DIR is the same as AIRFLOW_SOURCES. When jumping back-and-forth between ci/_utils.sh, breeze, and ci/ci_*.sh it can be confusing to keep track of what is what. This changes `breeze` to use `AIRFLOW_SOURCES` to referrer to the top level folder instead -- that means I don't have to keep as much context in my head
2020-06-02 09:39:42 +03:00
"${AIRFLOW_SOURCES}/confirm" "Proceeding with the initialization"
[AIRFLOW-6820] split breeze into functions (#7433)
2020-02-19 03:58:53 +03:00
echo
Don't reuse MY_DIR in breeze to mean different folder from ci/_utils.sh (#9098) scripts/ci/*.sh uses MY_DIR to mean scripts/ci, but in `breeze` MY_DIR is the same as AIRFLOW_SOURCES. When jumping back-and-forth between ci/_utils.sh, breeze, and ci/ci_*.sh it can be confusing to keep track of what is what. This changes `breeze` to use `AIRFLOW_SOURCES` to referrer to the top level folder instead -- that means I don't have to keep as much context in my head
2020-06-02 09:39:42 +03:00
pushd "${AIRFLOW_SOURCES}"
[AIRFLOW-6820] split breeze into functions (#7433)
2020-02-19 03:58:53 +03:00
set +e
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
# We need to export this one to speed up Cassandra driver installation in virtualenv
CASS_DRIVER_NO_CYTHON="1" pip install -e ".[devel]" \
Constraint files are now maintained automatically (#9889) * Constraint files are now maintained automatically * No need to generate requirements when setup.py changes * requirements are kept in separate orphan branches not in main repo * merges to master verify if latest requirements are working and push tested requirements to orphaned branches * we keep history of requirement changes and can label them individually for each version (by constraint-1.10.n tag name) * consistently changed all references to be 'constraints' not 'requirements'
2020-07-20 15:36:03 +03:00
--constraint "https://raw.githubusercontent.com/apache/airflow/${DEFAULT_CONSTRAINTS_BRANCH}/constraints-${PYTHON_MAJOR_MINOR_VERSION}.txt"
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
res=$?
[AIRFLOW-6820] split breeze into functions (#7433)
2020-02-19 03:58:53 +03:00
set -e
popd
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
if [[ ${res} != "0" ]]; then
[AIRFLOW-6820] split breeze into functions (#7433)
2020-02-19 03:58:53 +03:00
echo "#######################################################################"
echo " You had some troubles installing the venv !!!!!"
Help for breeze commands contain relevant flags. (#8261) * Help for breeze commands contain relevant flags. * fixup! Help for breeze commands contain relevant flags. * fixup! fixup! Help for breeze commands contain relevant flags.
2020-04-13 09:51:33 +03:00
echo " Try running the command below and rerun virtualenv installation"
[AIRFLOW-6820] split breeze into functions (#7433)
2020-02-19 03:58:53 +03:00
echo
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
if [[ ${OSTYPE} == "darwin"* ]]; then
Help for breeze commands contain relevant flags. (#8261) * Help for breeze commands contain relevant flags. * fixup! Help for breeze commands contain relevant flags. * fixup! fixup! Help for breeze commands contain relevant flags.
2020-04-13 09:51:33 +03:00
echo " brew install sqlite mysql postgresql openssl"
[AIRFLOW-6820] split breeze into functions (#7433)
2020-02-19 03:58:53 +03:00
else
Add 3.8 to the test matrices (#8836)
2020-06-05 20:39:28 +03:00
echo " sudo apt install build-essentials python3.6-dev python3.7-dev python3.8-dev python-dev openssl \\"
[AIRFLOW-6838] Introduce real subcommands for Breeze (#7515) This change introduces sub-commands in breeze tool. It is much needed as we have many commands now and it was difficult to separate commands from flags. Also --help output was very long and unreadable. With this change help it is much easier to discover what breeze can do for you as well as navigate with it. Co-authored-by: Jarek Potiuk <jarek@potiuk.com> Co-authored-by: Kamil Breguła <mik-laj@users.noreply.github.com>
2020-02-25 00:31:50 +03:00
echo " sqlite sqlite-dev default-libmysqlclient-dev libmysqld-dev postgresql"
[AIRFLOW-6820] split breeze into functions (#7433)
2020-02-19 03:58:53 +03:00
fi
echo
echo "#######################################################################"
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
exit ${res}
[AIRFLOW-6820] split breeze into functions (#7433)
2020-02-19 03:58:53 +03:00
fi
echo
echo "Wiping and recreating ${AIRFLOW_HOME_DIR}"
echo
rm -rvf "${AIRFLOW_HOME_DIR}"
mkdir -p "${AIRFLOW_HOME_DIR}"
echo
echo "Resetting AIRFLOW sqlite database"
echo
Remove errors raised during initialiation of virtualenv (#10896) Our migration scripts are written in the way that if you are running them to reset the db and you have some files in example_dags or dags, or some plugins in your plugins folder, you will get various kinds of errors. Those errors are printed to the user and even if they are ultimately ignored by the SqlAlchemy migration process, they are confusing whether the database reset (and thus the whole initialize-virtualenv) were successful or not. This change disables any DAGs that could be loaded during the DB reset (in case of running './breeze initialize-virtualenv'. Fixes #10894
2020-09-12 15:39:34 +03:00
AIRFLOW__CORE__LOAD_EXAMPLES="False" \
AIRFLOW__CORE__UNIT_TEST_MODE="False" \
AIRFLOW__CORE__SQL_ALCHEMY_POOL_ENABLED="False" \
AIRFLOW__CORE__DAGS_FOLDER="${AIRFLOW_SOURCES}/empty" \
AIRFLOW__CORE__PLUGINS_FOLDER="${AIRFLOW_SOURCES}/empty" \
airflow db reset -y
[AIRFLOW-6820] split breeze into functions (#7433)
2020-02-19 03:58:53 +03:00
echo
echo "Resetting AIRFLOW sqlite unit test database"
echo
Remove errors raised during initialiation of virtualenv (#10896) Our migration scripts are written in the way that if you are running them to reset the db and you have some files in example_dags or dags, or some plugins in your plugins folder, you will get various kinds of errors. Those errors are printed to the user and even if they are ultimately ignored by the SqlAlchemy migration process, they are confusing whether the database reset (and thus the whole initialize-virtualenv) were successful or not. This change disables any DAGs that could be loaded during the DB reset (in case of running './breeze initialize-virtualenv'. Fixes #10894
2020-09-12 15:39:34 +03:00
AIRFLOW__CORE__LOAD_EXAMPLES="False" \
AIRFLOW__CORE__UNIT_TEST_MODE="True" \
AIRFLOW__CORE__SQL_ALCHEMY_POOL_ENABLED="False" \
AIRFLOW__CORE__DAGS_FOLDER="${AIRFLOW_SOURCES}/empty" \
AIRFLOW__CORE__PLUGINS_FOLDER="${AIRFLOW_SOURCES}/empty" \
airflow db reset -y
echo
echo "Initialization of virtualenv was successful! Go ahead and develop Airflow!"
echo
[AIRFLOW-6820] split breeze into functions (#7433)
2020-02-19 03:58:53 +03:00
exit 0
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
fi
[AIRFLOW-6820] split breeze into functions (#7433)
2020-02-19 03:58:53 +03:00
}
[AIRFLOW-3611] Simplified development environment (#4932)
2019-08-27 21:39:36 +03:00
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
#######################################################################################################
#
Add comments to breeze scripts (#8797)
2020-05-10 20:55:40 +03:00
# Sets up autocomplete for Breeze for both - bash and zsh
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
#
# Used globals:
#
# AIRFLOW_SOURCES
# HOME
# OSTYPE
#
#######################################################################################################
Add packages to function names in bash (#10670) (#10696) Inspired by the Google Shell Guide where they mentioned separating package names with :: I realized that this was one of the missing pieces in the bash scripts of ours. While we already had packages (in libraries folders) it's been difficult to realise which function is where. With introducing packages - equal to the library file name we are *almost* at a level of a structured language - and it's easier to find the functions if you are looking for them. Way easier in fact. Part of #10576 (cherry picked from commit cc551ba793344800d2d396c13d7fd0c8eed97352) (cherry picked from commit 2bba276f0f06a5981bdd7e4f0e7e5ca2fe84f063)
2020-09-02 22:58:37 +03:00
function breeze::setup_autocomplete() {
[AIRFLOW-6820] split breeze into functions (#7433)
2020-02-19 03:58:53 +03:00
echo "Installing bash/zsh completion for local user"
echo
Don't reuse MY_DIR in breeze to mean different folder from ci/_utils.sh (#9098) scripts/ci/*.sh uses MY_DIR to mean scripts/ci, but in `breeze` MY_DIR is the same as AIRFLOW_SOURCES. When jumping back-and-forth between ci/_utils.sh, breeze, and ci/ci_*.sh it can be confusing to keep track of what is what. This changes `breeze` to use `AIRFLOW_SOURCES` to referrer to the top level folder instead -- that means I don't have to keep as much context in my head
2020-06-02 09:39:42 +03:00
"${AIRFLOW_SOURCES}/confirm" "This will create ~/.bash_completion.d/ directory and modify ~/.*rc files"
[AIRFLOW-6820] split breeze into functions (#7433)
2020-02-19 03:58:53 +03:00
echo
echo
mkdir -pv ~/.bash_completion.d
Don't reuse MY_DIR in breeze to mean different folder from ci/_utils.sh (#9098) scripts/ci/*.sh uses MY_DIR to mean scripts/ci, but in `breeze` MY_DIR is the same as AIRFLOW_SOURCES. When jumping back-and-forth between ci/_utils.sh, breeze, and ci/ci_*.sh it can be confusing to keep track of what is what. This changes `breeze` to use `AIRFLOW_SOURCES` to referrer to the top level folder instead -- that means I don't have to keep as much context in my head
2020-06-02 09:39:42 +03:00
ln -sf "${AIRFLOW_SOURCES}/breeze-complete" "${HOME}/.bash_completion.d/"
More robust and re-runnable autocomplete setup in Breeze (#9685) * More robust and re-runnable autocomplete setup in Breeze * Update breeze Co-authored-by: Tomek Urbaszek <turbaszek@gmail.com> Co-authored-by: Tomek Urbaszek <turbaszek@gmail.com>
2020-07-06 13:14:11 +03:00
echo
echo "Breeze Bash completion is now linked to: ${AIRFLOW_SOURCES}/breeze-complete"
echo
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
local breeze_comment="Added by Airflow Breeze autocomplete setup"
if ! grep "${breeze_comment}" "${HOME}/.bashrc" >/dev/null 2>&1; then
More robust and re-runnable autocomplete setup in Breeze (#9685) * More robust and re-runnable autocomplete setup in Breeze * Update breeze Co-authored-by: Tomek Urbaszek <turbaszek@gmail.com> Co-authored-by: Tomek Urbaszek <turbaszek@gmail.com>
2020-07-06 13:14:11 +03:00
touch ~/.bashrc
# shellcheck disable=SC2129
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
echo "# START: ${breeze_comment}" >>~/.bashrc
More robust and re-runnable autocomplete setup in Breeze (#9685) * More robust and re-runnable autocomplete setup in Breeze * Update breeze Co-authored-by: Tomek Urbaszek <turbaszek@gmail.com> Co-authored-by: Tomek Urbaszek <turbaszek@gmail.com>
2020-07-06 13:14:11 +03:00
cat <<"EOF" >>~/.bashrc
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
for bcfile in ~/.bash_completion.d/* ; do
. ${bcfile}
[AIRFLOW-6820] split breeze into functions (#7433)
2020-02-19 03:58:53 +03:00
done
[AIRFLOW-3611] Simplified development environment (#4932)
2019-08-27 21:39:36 +03:00
EOF
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
echo "# END: ${breeze_comment}" >>~/.bashrc
More robust and re-runnable autocomplete setup in Breeze (#9685) * More robust and re-runnable autocomplete setup in Breeze * Update breeze Co-authored-by: Tomek Urbaszek <turbaszek@gmail.com> Co-authored-by: Tomek Urbaszek <turbaszek@gmail.com>
2020-07-06 13:14:11 +03:00
echo
echo "The ${HOME}/.bashrc has been modified"
echo
else
echo
echo "The ${HOME}/.bashrc was already modified before. Not changing it."
echo
fi
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
if ! grep "${breeze_comment}" "${HOME}/.zshrc" >/dev/null 2>&1; then
More robust and re-runnable autocomplete setup in Breeze (#9685) * More robust and re-runnable autocomplete setup in Breeze * Update breeze Co-authored-by: Tomek Urbaszek <turbaszek@gmail.com> Co-authored-by: Tomek Urbaszek <turbaszek@gmail.com>
2020-07-06 13:14:11 +03:00
# shellcheck disable=SC2129
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
echo "# START: ${breeze_comment}" >>~/.zshrc
More robust and re-runnable autocomplete setup in Breeze (#9685) * More robust and re-runnable autocomplete setup in Breeze * Update breeze Co-authored-by: Tomek Urbaszek <turbaszek@gmail.com> Co-authored-by: Tomek Urbaszek <turbaszek@gmail.com>
2020-07-06 13:14:11 +03:00
cat <<"EOF" >>~/.zshrc
[AIRFLOW-6820] split breeze into functions (#7433)
2020-02-19 03:58:53 +03:00
autoload compinit && compinit
autoload bashcompinit && bashcompinit
source ~/.bash_completion.d/breeze-complete
EOF
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
echo "# END: ${breeze_comment}" >>~/.zshrc
More robust and re-runnable autocomplete setup in Breeze (#9685) * More robust and re-runnable autocomplete setup in Breeze * Update breeze Co-authored-by: Tomek Urbaszek <turbaszek@gmail.com> Co-authored-by: Tomek Urbaszek <turbaszek@gmail.com>
2020-07-06 13:14:11 +03:00
echo
echo "The ${HOME}/.zshrc has been modified"
echo
else
echo
echo "The ${HOME}/.zshrc was already modified before. Not changing it."
echo
fi
[AIRFLOW-6820] split breeze into functions (#7433)
2020-02-19 03:58:53 +03:00
if [[ "${OSTYPE}" == "darwin"* ]]; then
# For MacOS we have to handle the special case where terminal app DOES NOT run .bashrc by default
# But re-runs .bash_profile :(
# See https://scriptingosx.com/2017/04/about-bash_profile-and-bashrc-on-macos/
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
if ! grep "${breeze_comment}" "${HOME}/.bash_profile"; then
More robust and re-runnable autocomplete setup in Breeze (#9685) * More robust and re-runnable autocomplete setup in Breeze * Update breeze Co-authored-by: Tomek Urbaszek <turbaszek@gmail.com> Co-authored-by: Tomek Urbaszek <turbaszek@gmail.com>
2020-07-06 13:14:11 +03:00
# shellcheck disable=SC2129
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
echo "# START: ${breeze_comment}" >>~/.bash_profile
More robust and re-runnable autocomplete setup in Breeze (#9685) * More robust and re-runnable autocomplete setup in Breeze * Update breeze Co-authored-by: Tomek Urbaszek <turbaszek@gmail.com> Co-authored-by: Tomek Urbaszek <turbaszek@gmail.com>
2020-07-06 13:14:11 +03:00
cat <<"EOF" >>~/.bash_profile
[AIRFLOW-6820] split breeze into functions (#7433)
2020-02-19 03:58:53 +03:00
if [ -r ~/.bashrc ]; then
source ~/.bashrc
fi
EOF
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
echo "# END: ${breeze_comment}" >>~/.bash_profile
More robust and re-runnable autocomplete setup in Breeze (#9685) * More robust and re-runnable autocomplete setup in Breeze * Update breeze Co-authored-by: Tomek Urbaszek <turbaszek@gmail.com> Co-authored-by: Tomek Urbaszek <turbaszek@gmail.com>
2020-07-06 13:14:11 +03:00
echo
echo "The ${HOME}/.bash_profile has been modified"
echo
else
echo
echo "The ${HOME}/.bash_profile was already modified before. Not changing it."
echo
[AIRFLOW-6820] split breeze into functions (#7433)
2020-02-19 03:58:53 +03:00
fi
fi
echo
echo
More robust and re-runnable autocomplete setup in Breeze (#9685) * More robust and re-runnable autocomplete setup in Breeze * Update breeze Co-authored-by: Tomek Urbaszek <turbaszek@gmail.com> Co-authored-by: Tomek Urbaszek <turbaszek@gmail.com>
2020-07-06 13:14:11 +03:00
echo "Breeze completion is installed to ~/.bash_completion.d/breeze-complete"
[AIRFLOW-6820] split breeze into functions (#7433)
2020-02-19 03:58:53 +03:00
echo
More robust and re-runnable autocomplete setup in Breeze (#9685) * More robust and re-runnable autocomplete setup in Breeze * Update breeze Co-authored-by: Tomek Urbaszek <turbaszek@gmail.com> Co-authored-by: Tomek Urbaszek <turbaszek@gmail.com>
2020-07-06 13:14:11 +03:00
echo "Please exit and re-enter your shell or run:"
[AIRFLOW-6820] split breeze into functions (#7433)
2020-02-19 03:58:53 +03:00
echo
More robust and re-runnable autocomplete setup in Breeze (#9685) * More robust and re-runnable autocomplete setup in Breeze * Update breeze Co-authored-by: Tomek Urbaszek <turbaszek@gmail.com> Co-authored-by: Tomek Urbaszek <turbaszek@gmail.com>
2020-07-06 13:14:11 +03:00
echo " source ~/.bash_completion.d/breeze-complete"
[AIRFLOW-6820] split breeze into functions (#7433)
2020-02-19 03:58:53 +03:00
echo
exit 0
}
[AIRFLOW-3611] Simplified development environment (#4932)
2019-08-27 21:39:36 +03:00
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
#######################################################################################################
#
Add comments to breeze scripts (#8797)
2020-05-10 20:55:40 +03:00
# Prints information about the current configuration of Breeze - if you enter breeze interactively
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
# and you did not suppress cheatsheet or asciiart, it also prints those. It also prints values
Add packages to function names in bash (#10670) (#10696) Inspired by the Google Shell Guide where they mentioned separating package names with :: I realized that this was one of the missing pieces in the bash scripts of ours. While we already had packages (in libraries folders) it's been difficult to realise which function is where. With introducing packages - equal to the library file name we are *almost* at a level of a structured language - and it's easier to find the functions if you are looking for them. Way easier in fact. Part of #10576 (cherry picked from commit cc551ba793344800d2d396c13d7fd0c8eed97352) (cherry picked from commit 2bba276f0f06a5981bdd7e4f0e7e5ca2fe84f063)
2020-09-02 22:58:37 +03:00
# of constants set by breeze::read_saved_environment_variables() function and other initialization functions.
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
#
# Used globals:
#
# BACKEND
# POSTGRES_VERSION
# MYSQL_VERSION
# SUPPRESS_CHEATSHEET_FILE
# SUPPRESS_ASCIIART_FILE
# PRODUCTION_IMAGE
# BRANCH_NAME
# AIRFLOW_CI_IMAGE
# AIRFLOW_PROD_IMAGE
# AIRFLOW_VERSION
# DOCKERHUB_USER
# DOCKERHUB_REPO
# INSTALL_AIRFLOW_VERSION
# INSTALL_AIRFLOW_REFERENCE
#
# Outputs:
# Prints the information about the build to stdout.
#
#######################################################################################################
Add packages to function names in bash (#10670) (#10696) Inspired by the Google Shell Guide where they mentioned separating package names with :: I realized that this was one of the missing pieces in the bash scripts of ours. While we already had packages (in libraries folders) it's been difficult to realise which function is where. With introducing packages - equal to the library file name we are *almost* at a level of a structured language - and it's easier to find the functions if you are looking for them. Way easier in fact. Part of #10576 (cherry picked from commit cc551ba793344800d2d396c13d7fd0c8eed97352) (cherry picked from commit 2bba276f0f06a5981bdd7e4f0e7e5ca2fe84f063)
2020-09-02 22:58:37 +03:00
function breeze::print_badge() {
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
local backend_version=""
[AIRFLOW-7058] Add support for different DB versions (#7717)
2020-03-15 01:57:47 +03:00
if [[ ${BACKEND} == "postgres" ]]; then
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
backend_version="${POSTGRES_VERSION}"
[AIRFLOW-7058] Add support for different DB versions (#7717)
2020-03-15 01:57:47 +03:00
elif [[ ${BACKEND} == "mysql" ]]; then
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
backend_version="${MYSQL_VERSION}"
[AIRFLOW-7058] Add support for different DB versions (#7717)
2020-03-15 01:57:47 +03:00
fi
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
if [[ ! -f "${SUPPRESS_ASCIIART_FILE}" && ${command_to_run} == "enter_breeze" ]]; then
[AIRFLOW-3611] Simplified development environment (#4932)
2019-08-27 21:39:36 +03:00
cat <<EOF
@&&&&&&@
@&&&&&&&&&&&@
&&&&&&&&&&&&&&&&
&&&&&&&&&&
&&&&&&&
&&&&&&&
@@@@@@@@@@@@@@@@ &&&&&&
@&&&&&&&&&&&&&&&&&&&&&&&&&&
&&&&&&&&&&&&&&&&&&&&&&&&&&&&
&&&&&&&&&&&&
&&&&&&&&&
&&&&&&&&&&&&
@@&&&&&&&&&&&&&&&@
@&&&&&&&&&&&&&&&&&&&&&&&&&&&& &&&&&&
&&&&&&&&&&&&&&&&&&&&&&&&&&&& &&&&&&
&&&&&&&&&&&&&&&&&&&&&&&& &&&&&&
&&&&&&
&&&&&&&
@&&&&&&&&
@&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&
&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&
&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&
@&&&@ && @&&&&&&&&&&& &&&&&&&&&&&& && &&&&&&&&&& &&& &&& &&&
&&& &&& && @&& &&& && && &&& &&&@ &&& &&&&& &&&
&&& &&& && @&&&&&&&&&&&& &&&&&&&&&&& && && &&& &&& &&& &&@ &&&
&&&&&&&&&&& && @&&&&&&&&& && && &&@ &&& &&@&& &&@&&
&&& &&& && @&& &&&@ && &&&&&&&&&&& &&&&&&&&&&&& &&&& &&&&
&&&&&&&&&&&& &&&&&&&&&&&& &&&&&&&&&&&@ &&&&&&&&&&&& &&&&&&&&&&& &&&&&&&&&&&
&&& &&& && &&& && &&& &&&& &&
&&&&&&&&&&&&@ &&&&&&&&&&&& &&&&&&&&&&& &&&&&&&&&&& &&&& &&&&&&&&&&
&&& && && &&&& && &&& &&&& &&
&&&&&&&&&&&&& && &&&&@ &&&&&&&&&&&@ &&&&&&&&&&&& @&&&&&&&&&&& &&&&&&&&&&&
Add Production Docker image support (#7832)
2020-04-02 20:52:11 +03:00
EOF
if [[ ${PRODUCTION_IMAGE} == "true" ]]; then
cat <<EOF
Use production image.
Branch name: ${BRANCH_NAME}
Docker image: ${AIRFLOW_PROD_IMAGE}
Add packages to function names in bash (#10670) (#10696) Inspired by the Google Shell Guide where they mentioned separating package names with :: I realized that this was one of the missing pieces in the bash scripts of ours. While we already had packages (in libraries folders) it's been difficult to realise which function is where. With introducing packages - equal to the library file name we are *almost* at a level of a structured language - and it's easier to find the functions if you are looking for them. Way easier in fact. Part of #10576 (cherry picked from commit cc551ba793344800d2d396c13d7fd0c8eed97352) (cherry picked from commit 2bba276f0f06a5981bdd7e4f0e7e5ca2fe84f063)
2020-09-02 22:58:37 +03:00
Airflow source version: $(build_images::get_airflow_version_from_production_image)
Add Production Docker image support (#7832)
2020-04-02 20:52:11 +03:00
EOF
else
cat <<EOF
Use CI image.
[AIRFLOW-3611] Simplified development environment (#4932)
2019-08-27 21:39:36 +03:00
[AIRFLOW-6763] Make systems tests ready for backport tests (#7389) We will run system test on back-ported operators for 1.10* series of airflow and for that we need to have support for running system tests using pytest's markers and reading environment variables passed from HOST machine (to pass credentials).  This is the first step to automate system tests execution.
2020-02-21 20:25:32 +03:00
Branch name: ${BRANCH_NAME}
Docker image: ${AIRFLOW_CI_IMAGE}
Proper version is displayed when running prod image via Breeze (#8229)
2020-04-11 12:37:06 +03:00
Airflow source version: ${AIRFLOW_VERSION}
Add Production Docker image support (#7832)
2020-04-02 20:52:11 +03:00
EOF
fi
cat <<EOF
Requirements now depend on python version (#7841)
2020-03-27 17:27:53 +03:00
Python version: ${PYTHON_MAJOR_MINOR_VERSION}
[AIRFLOW-6763] Make systems tests ready for backport tests (#7389) We will run system test on back-ported operators for 1.10* series of airflow and for that we need to have support for running system tests using pytest's markers and reading environment variables passed from HOST machine (to pass credentials).  This is the first step to automate system tests execution.
2020-02-21 20:25:32 +03:00
DockerHub user: ${DOCKERHUB_USER}
DockerHub repo: ${DOCKERHUB_REPO}
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
Backend: ${BACKEND} ${backend_version}
[AIRFLOW-3611] Simplified development environment (#4932)
2019-08-27 21:39:36 +03:00
EOF
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
if [[ -n ${INSTALL_AIRFLOW_VERSION=} || -n ${INSTALL_AIRFLOW_REFERENCE=} ]]; then
Add Production Docker image support (#7832)
2020-04-02 20:52:11 +03:00
cat <<EOF
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
Airflow installed: ${INSTALL_AIRFLOW_VERSION=}${INSTALL_AIRFLOW_REFERENCE=}
Add Production Docker image support (#7832)
2020-04-02 20:52:11 +03:00
[AIRFLOW-3611] Simplified development environment (#4932)
2019-08-27 21:39:36 +03:00
EOF
fi
else
Add Production Docker image support (#7832)
2020-04-02 20:52:11 +03:00
if [[ ${PRODUCTION_IMAGE} == "true" ]]; then
cat <<EOF
More sensible docker caching strategy for Prod images (#9547) Local caching is now default strategy when building the Production image. You can still change it to pulled - similar to CI builds by providing the right build flag and this is what is used in CI by default. The flags in Breeze are now updated to be more eplanatory and friendly (build-cache-*) and a flag for "disabled" cache option is added as well. Also the Dockerfile and Dockerfile.ci files are not needed any more in the docker context. They used to be needed when we built the Kubernetes image in the container, but since we are now using production image directly - we do not need them any nmore. Combining setting the default strategy to local and removing the Dockerfile from the context has the nice effect that you can iterate much faster on the Production image without triggering rebuilds of half of the docker image as soon as the Dockerfile changes.
2020-06-28 18:38:17 +03:00
Production image.
Add Production Docker image support (#7832)
2020-04-02 20:52:11 +03:00
Branch name: ${BRANCH_NAME}
Docker image: ${AIRFLOW_PROD_IMAGE}
EOF
else
cat <<EOF
More sensible docker caching strategy for Prod images (#9547) Local caching is now default strategy when building the Production image. You can still change it to pulled - similar to CI builds by providing the right build flag and this is what is used in CI by default. The flags in Breeze are now updated to be more eplanatory and friendly (build-cache-*) and a flag for "disabled" cache option is added as well. Also the Dockerfile and Dockerfile.ci files are not needed any more in the docker context. They used to be needed when we built the Kubernetes image in the container, but since we are now using production image directly - we do not need them any nmore. Combining setting the default strategy to local and removing the Dockerfile from the context has the nice effect that you can iterate much faster on the Production image without triggering rebuilds of half of the docker image as soon as the Dockerfile changes.
2020-06-28 18:38:17 +03:00
CI image.
[AIRFLOW-3611] Simplified development environment (#4932)
2019-08-27 21:39:36 +03:00
[AIRFLOW-6763] Make systems tests ready for backport tests (#7389) We will run system test on back-ported operators for 1.10* series of airflow and for that we need to have support for running system tests using pytest's markers and reading environment variables passed from HOST machine (to pass credentials).  This is the first step to automate system tests execution.
2020-02-21 20:25:32 +03:00
Branch name: ${BRANCH_NAME}
Docker image: ${AIRFLOW_CI_IMAGE}
Add Production Docker image support (#7832)
2020-04-02 20:52:11 +03:00
EOF
fi
cat <<EOF
[AIRFLOW-6763] Make systems tests ready for backport tests (#7389) We will run system test on back-ported operators for 1.10* series of airflow and for that we need to have support for running system tests using pytest's markers and reading environment variables passed from HOST machine (to pass credentials).  This is the first step to automate system tests execution.
2020-02-21 20:25:32 +03:00
Airflow source version: ${AIRFLOW_VERSION}
Requirements now depend on python version (#7841)
2020-03-27 17:27:53 +03:00
Python version: ${PYTHON_MAJOR_MINOR_VERSION}
[AIRFLOW-6763] Make systems tests ready for backport tests (#7389) We will run system test on back-ported operators for 1.10* series of airflow and for that we need to have support for running system tests using pytest's markers and reading environment variables passed from HOST machine (to pass credentials).  This is the first step to automate system tests execution.
2020-02-21 20:25:32 +03:00
DockerHub user: ${DOCKERHUB_USER}
DockerHub repo: ${DOCKERHUB_REPO}
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
Backend: ${BACKEND} ${backend_version}
[AIRFLOW-3611] Simplified development environment (#4932)
2019-08-27 21:39:36 +03:00
EOF
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
if [[ -n ${INSTALL_AIRFLOW_VERSION=} || -n ${INSTALL_AIRFLOW_REFERENCE=} ]]; then
Add Production Docker image support (#7832)
2020-04-02 20:52:11 +03:00
cat <<EOF
Airflow installed from: ${INSTALL_AIRFLOW_VERSION}${INSTALL_AIRFLOW_REFERENCE}
EOF
fi
[AIRFLOW-3611] Simplified development environment (#4932)
2019-08-27 21:39:36 +03:00
fi
}
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
#######################################################################################################
#
# Prepares command file that can be used to easily run the docker commands outside of Breeze.
#
Add comments to breeze scripts (#8797)
2020-05-10 20:55:40 +03:00
# The command file generated in cache ./build directory is a standalone script that contains
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
# All the environment variables and docker-compose configuration to run the command.
# This is because depending on configuration of Breeze we might have different compose files
# used and different env variables set.
#
# Those are a convenience scripts that you might use to debug command execution although
# In most cases they are used internally by Breeze.
#
# Used Globals:
# BRANCH_NAME
# PYTHON_MAJOR_MINOR_VERSION
# DOCKERHUB_USER
# DOCKERHUB_REPO
# HOST_AIRFLOW_SOURCES
# BACKEND
# AIRFLOW_VERSION
# INSTALL_AIRFLOW_VERSION
# WEBSERVER_HOST_PORT
# POSTGRES_HOST_PORT
# POSTGRES_VERSION
# MYSQL_HOST_PORT
# MYSQL_VERSION
# AIRFLOW_SOURCES
# AIRFLOW_CI_IMAGE
# AIRFLOW_PROD_IMAGE
# AIRFLOW_PROD_IMAGE_KUBERNETES
# AIRFLOW_PROD_BASE_TAG
# SQLITE_URL
#
# Arguments:
#
# file to prepare
# command to run
# compose_file to use
# airflow_image to use
#
# Outputs:
# Creates the convenience command file that can be run to use the docker command.
#
#######################################################################################################
Add packages to function names in bash (#10670) (#10696) Inspired by the Google Shell Guide where they mentioned separating package names with :: I realized that this was one of the missing pieces in the bash scripts of ours. While we already had packages (in libraries folders) it's been difficult to realise which function is where. With introducing packages - equal to the library file name we are *almost* at a level of a structured language - and it's easier to find the functions if you are looking for them. Way easier in fact. Part of #10576 (cherry picked from commit cc551ba793344800d2d396c13d7fd0c8eed97352) (cherry picked from commit 2bba276f0f06a5981bdd7e4f0e7e5ca2fe84f063)
2020-09-02 22:58:37 +03:00
function breeze::prepare_command_file() {
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
local file="${1}"
local command="${2}"
local compose_file="${3}"
local airflow_image="${4}"
cat <<EOF >"${file}"
[AIRFLOW-3611] Simplified development environment (#4932)
2019-08-27 21:39:36 +03:00
#!/usr/bin/env bash
Improve production image iteration speed (#9162) For a long time the way how entrypoint worked in ci scripts was wrong. The way it worked was convoluted and short of black magic. This did not allow to pass multiple test targets and required separate execute command scripts in Breeze. This is all now straightened out and both production and CI image are always using the right entrypoint by default and we can simply pass parameters to the image as usual without escaping strings. This also allowed to remove some breeze commands and change names of several flags in Breeze to make them more meaningful. Both CI and PROD image have now embedded scripts for log cleaning. History of image releases is added for 1.10.10-* alpha quality images.
2020-06-16 13:36:46 +03:00
if [[ \${VERBOSE} == "true" ]]; then
echo
echo "Executing script:"
echo
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
echo "${file} \${@}"
Improve production image iteration speed (#9162) For a long time the way how entrypoint worked in ci scripts was wrong. The way it worked was convoluted and short of black magic. This did not allow to pass multiple test targets and required separate execute command scripts in Breeze. This is all now straightened out and both production and CI image are always using the right entrypoint by default and we can simply pass parameters to the image as usual without escaping strings. This also allowed to remove some breeze commands and change names of several flags in Breeze to make them more meaningful. Both CI and PROD image have now embedded scripts for log cleaning. History of image releases is added for 1.10.10-* alpha quality images.
2020-06-16 13:36:46 +03:00
echo
set -x
fi
cd "\$( dirname "\${BASH_SOURCE[0]}" )" || exit
[AIRFLOW-3611] Simplified development environment (#4932)
2019-08-27 21:39:36 +03:00
export DOCKERHUB_USER=${DOCKERHUB_USER}
export DOCKERHUB_REPO=${DOCKERHUB_REPO}
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
export HOST_USER_ID=${HOST_USER_ID}
export HOST_GROUP_ID=${HOST_GROUP_ID}
Improve production image iteration speed (#9162) For a long time the way how entrypoint worked in ci scripts was wrong. The way it worked was convoluted and short of black magic. This did not allow to pass multiple test targets and required separate execute command scripts in Breeze. This is all now straightened out and both production and CI image are always using the right entrypoint by default and we can simply pass parameters to the image as usual without escaping strings. This also allowed to remove some breeze commands and change names of several flags in Breeze to make them more meaningful. Both CI and PROD image have now embedded scripts for log cleaning. History of image releases is added for 1.10.10-* alpha quality images.
2020-06-16 13:36:46 +03:00
export HOST_AIRFLOW_SOURCES="${AIRFLOW_SOURCES}"
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
export COMPOSE_FILE="${compose_file}"
Requirements now depend on python version (#7841)
2020-03-27 17:27:53 +03:00
export PYTHON_MAJOR_MINOR_VERSION="${PYTHON_MAJOR_MINOR_VERSION}"
[AIRFLOW-3611] Simplified development environment (#4932)
2019-08-27 21:39:36 +03:00
export BACKEND="${BACKEND}"
export AIRFLOW_VERSION="${AIRFLOW_VERSION}"
[AIRFLOW-6763] Make systems tests ready for backport tests (#7389) We will run system test on back-ported operators for 1.10* series of airflow and for that we need to have support for running system tests using pytest's markers and reading environment variables passed from HOST machine (to pass credentials).  This is the first step to automate system tests execution.
2020-02-21 20:25:32 +03:00
export INSTALL_AIRFLOW_VERSION="${INSTALL_AIRFLOW_VERSION}"
[AIRFLOW-3611] Simplified development environment (#4932)
2019-08-27 21:39:36 +03:00
export WEBSERVER_HOST_PORT="${WEBSERVER_HOST_PORT}"
export POSTGRES_HOST_PORT="${POSTGRES_HOST_PORT}"
[AIRFLOW-7058] Add support for different DB versions (#7717)
2020-03-15 01:57:47 +03:00
export POSTGRES_VERSION="${POSTGRES_VERSION}"
[AIRFLOW-3611] Simplified development environment (#4932)
2019-08-27 21:39:36 +03:00
export MYSQL_HOST_PORT="${MYSQL_HOST_PORT}"
[AIRFLOW-7058] Add support for different DB versions (#7717)
2020-03-15 01:57:47 +03:00
export MYSQL_VERSION="${MYSQL_VERSION}"
Improved cloud tool available in the trimmed down CI container (#9167) * Improved cloud tool available in the trimmed down CI container The tools now have shebangs which make them available for python tools. Also /opt/airflow is now mounted from the host Airflow sources which makes it possible for the tools to copy files directly to/from the sources of Airflow. It also contains one small change for Linux users - the files created by docker gcloud are created with root user so in order to fix that the directories mounted from the host are fixed when you exit the tool - their ownership is changed to be owned by the host user
2020-06-09 10:33:16 +03:00
export AIRFLOW_SOURCES="${AIRFLOW_SOURCES}"
[AIRFLOW-5704] Improve Kind Kubernetes scripts for local testing (#6516) * Fixed problem that Kubernetes tests were testing latest master rather than what came from the local sources. * Kind (Kubernetes in Dcocker) is run in the same Docker as Breeze env * Moved Kubernetes scripts to 'in_container' dir where they belong now * Kubernetes cluster is reused until it is stopped * Kubernetes image is build from image in docker already + mounted sources * Kubectl version name is corrected in the Dockerfile * KUBERNETES_VERSION can now be used to select Kubernetes version * Running kubernetes scripts is now easy in Breeze * We can start/recreate/stop cluster using --<ACTION>-kind-cluster * Instructions on how to run Kubernetes tests are updated * The old "bare" environment is replaced by --no-deps switch
2020-01-11 18:25:19 +03:00
export AIRFLOW_CI_IMAGE="${AIRFLOW_CI_IMAGE}"
Improve production image iteration speed (#9162) For a long time the way how entrypoint worked in ci scripts was wrong. The way it worked was convoluted and short of black magic. This did not allow to pass multiple test targets and required separate execute command scripts in Breeze. This is all now straightened out and both production and CI image are always using the right entrypoint by default and we can simply pass parameters to the image as usual without escaping strings. This also allowed to remove some breeze commands and change names of several flags in Breeze to make them more meaningful. Both CI and PROD image have now embedded scripts for log cleaning. History of image releases is added for 1.10.10-* alpha quality images.
2020-06-16 13:36:46 +03:00
export AIRFLOW_PROD_IMAGE="${AIRFLOW_PROD_IMAGE}"
CI Images are now pre-build and stored in registry (#10368) * CI Images are now pre-build and stored in registry With this change we utilise the latest pull_request_target event type from Github Actions and we are building the CI image only once (per version) for the entire run. This safes from 2 to 10 minutes per job (!) depending on how much of the Docker image needs to be rebuilt. It works in the way that the image is built only in the build-or-wait step. In case of direct push run or scheduled runs, the build-or-wait step builds and pushes to the GitHub registry the CI image. In case of the pull_request runs, the build-and-wait step waits until separate build-ci-image.yml workflow builds and pushes the image and it will only move forward once the image is ready. This has numerous advantages: 1) Each job that requires CI image is much faster because instead of pulling + rebuilding the image it only pulls the image that was build once. This saves around 2 minutes per job in regular builds but in case of python patch level updates, or adding new requirements it can save up to 10 minutes per job (!) 2) While the images are buing rebuilt we only block one job waiting for all the images. The tests will start running in parallell only when all images are ready, so we are not blocking other runs from running. 3) Whole run uses THE SAME image. Previously we could have some variations because the images were built at different times and potentially releases of dependencies in-between several jobs could make different jobs in the same run use slightly different image. This is not happening any more. 4) Also when we push image to github or dockerhub we push the very same image that was built and tested. Previously it could happen that the image pushed was slightly different than the one that was used for testing (for the same reason) 5) Similar case is with the production images. We are now building and pushing consistently the same images accross the board. 6) Documentation building is split into two parallel jobs docs building and spell checking - decreases elapsed time for the docs build. 7) Last but not least - we keep the history of al the images - those images contain SHA of the commit. This means that we can simply download and run the image locally to reproduce any problem that anyone had in their PR (!). This is super useful to be able to help others to test their problems. * fixup! CI Images are now pre-build and stored in registry * fixup! fixup! CI Images are now pre-build and stored in registry * fixup! fixup! fixup! CI Images are now pre-build and stored in registry * fixup! fixup! fixup! CI Images are now pre-build and stored in registry
2020-08-20 10:57:07 +03:00
export AIRFLOW_PROD_IMAGE_KUBERNETES="${AIRFLOW_PROD_IMAGE_KUBERNETES}"
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
export AIRFLOW_PROD_BASE_TAG="${AIRFLOW_PROD_BASE_TAG}"
export AIRFLOW_IMAGE="${airflow_image}"
Add comments to breeze scripts (#8797)
2020-05-10 20:55:40 +03:00
export SQLITE_URL="${SQLITE_URL}"
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
docker-compose --log-level INFO ${command}
[AIRFLOW-3611] Simplified development environment (#4932)
2019-08-27 21:39:36 +03:00
EOF
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
chmod u+x "${file}"
[AIRFLOW-3611] Simplified development environment (#4932)
2019-08-27 21:39:36 +03:00
}
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
#######################################################################################################
#
# Prepare all command files that we are using. Depending on the command to execute we use two
Add comments to breeze scripts (#8797)
2020-05-10 20:55:40 +03:00
# convenience scripts:
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
#
Add comments to breeze scripts (#8797)
2020-05-10 20:55:40 +03:00
# dc_ci - to run docker compose command for CI image
# dc_prod - to run docker compose command for PROD image
#
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
# Global constants set:
#
# PYTHON_BASE_IMAGE_VERSION
# PYTHON_BASE_IMAGE
# AIRFLOW_CI_IMAGE
# AIRFLOW_PROD_BASE_TAG
# AIRFLOW_PROD_IMAGE
# AIRFLOW_PROD_IMAGE_KUBERNETES
# BUILT_CI_IMAGE_FLAG_FILE
#
#######################################################################################################
Add packages to function names in bash (#10670) (#10696) Inspired by the Google Shell Guide where they mentioned separating package names with :: I realized that this was one of the missing pieces in the bash scripts of ours. While we already had packages (in libraries folders) it's been difficult to realise which function is where. With introducing packages - equal to the library file name we are *almost* at a level of a structured language - and it's easier to find the functions if you are looking for them. Way easier in fact. Part of #10576 (cherry picked from commit cc551ba793344800d2d396c13d7fd0c8eed97352) (cherry picked from commit 2bba276f0f06a5981bdd7e4f0e7e5ca2fe84f063)
2020-09-02 22:58:37 +03:00
function breeze::prepare_command_files() {
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
local main_ci_docker_compose_file=${SCRIPTS_CI_DIR}/docker-compose/base.yml
local main_prod_docker_compose_file=${SCRIPTS_CI_DIR}/docker-compose/base.yml
local backend_docker_compose_file=${SCRIPTS_CI_DIR}/docker-compose/backend-${BACKEND}.yml
local local_docker_compose_file=${SCRIPTS_CI_DIR}/docker-compose/local.yml
local files_docker_compose_file=${SCRIPTS_CI_DIR}/docker-compose/files.yml
local local_prod_docker_compose_file=${SCRIPTS_CI_DIR}/docker-compose/local-prod.yml
local remove_sources_docker_compose_file=${SCRIPTS_CI_DIR}/docker-compose/remove-sources.yml
local forward_credentials_docker_compose_file=${SCRIPTS_CI_DIR}/docker-compose/forward-credentials.yml
local compose_ci_file=${main_ci_docker_compose_file}:${backend_docker_compose_file}
local compose_prod_file=${main_prod_docker_compose_file}:${backend_docker_compose_file}
[AIRFLOW-6820] split breeze into functions (#7433)
2020-02-19 03:58:53 +03:00
[AIRFLOW-5828] Move build logic out from hooks/build (#7618) This is the final step of simplifying the Breeze scripts by moving all the logic out from Travis' hooks/build
2020-03-07 18:09:06 +03:00
if [[ "${MOUNT_LOCAL_SOURCES}" != "false" ]]; then
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
compose_ci_file=${compose_ci_file}:${local_docker_compose_file}
compose_prod_file=${compose_prod_file}:${local_prod_docker_compose_file}
[AIRFLOW-6820] split breeze into functions (#7433)
2020-02-19 03:58:53 +03:00
fi
Replace file.io with artifacts (#10137)
2020-08-04 02:26:59 +03:00
if [[ "${MOUNT_FILES}" != "false" ]]; then
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
compose_ci_file=${compose_ci_file}:${files_docker_compose_file}
compose_prod_file=${compose_prod_file}:${files_docker_compose_file}
Replace file.io with artifacts (#10137)
2020-08-04 02:26:59 +03:00
fi
[AIRFLOW-6763] Make systems tests ready for backport tests (#7389) We will run system test on back-ported operators for 1.10* series of airflow and for that we need to have support for running system tests using pytest's markers and reading environment variables passed from HOST machine (to pass credentials).  This is the first step to automate system tests execution.
2020-02-21 20:25:32 +03:00
if [[ ${FORWARD_CREDENTIALS} == "true" ]]; then
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
compose_ci_file=${compose_ci_file}:${forward_credentials_docker_compose_file}
compose_prod_file=${compose_prod_file}:${forward_credentials_docker_compose_file}
[AIRFLOW-6763] Make systems tests ready for backport tests (#7389) We will run system test on back-ported operators for 1.10* series of airflow and for that we need to have support for running system tests using pytest's markers and reading environment variables passed from HOST machine (to pass credentials).  This is the first step to automate system tests execution.
2020-02-21 20:25:32 +03:00
fi
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
if [[ -n ${INSTALL_AIRFLOW_VERSION=} ]]; then
compose_ci_file=${compose_ci_file}:${remove_sources_docker_compose_file}
[AIRFLOW-6763] Make systems tests ready for backport tests (#7389) We will run system test on back-ported operators for 1.10* series of airflow and for that we need to have support for running system tests using pytest's markers and reading environment variables passed from HOST machine (to pass credentials).  This is the first step to automate system tests execution.
2020-02-21 20:25:32 +03:00
fi
[AIRFLOW-6820] split breeze into functions (#7433)
2020-02-19 03:58:53 +03:00
set +u
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
local unique_integrations
[AIRFLOW-6820] split breeze into functions (#7433)
2020-02-19 03:58:53 +03:00
# shellcheck disable=SC2207
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
unique_integrations=($(echo "${INTEGRATIONS[@]}" | tr ' ' '\n' | sort -u | tr '\n' ' '))
[AIRFLOW-6820] split breeze into functions (#7433)
2020-02-19 03:58:53 +03:00
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
local integration
for integration in "${unique_integrations[@]}"; do
compose_ci_file=${compose_ci_file}:${SCRIPTS_CI_DIR}/docker-compose/integration-${integration}.yml
[AIRFLOW-6820] split breeze into functions (#7433)
2020-02-19 03:58:53 +03:00
done
set -u
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
export DOCKER_COMPOSE_RUN_SCRIPT_FOR_CI="dc_ci"
readonly DOCKER_COMPOSE_RUN_SCRIPT_FOR_CI
Revert recent breeze changes (#10651 & #10670) (#10694) * Revert "Add packages to function names in bash (#10670)" This reverts commit cc551ba793344800d2d396c13d7fd0c8eed97352. * Revert "Implement Google Shell Conventions for breeze script … (#10651)" This reverts commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2.
2020-09-02 19:27:36 +03:00
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
export DOCKER_COMPOSE_RUN_SCRIPT_FOR_PROD="dc_prod"
readonly DOCKER_COMPOSE_RUN_SCRIPT_FOR_PROD
[AIRFLOW-6820] split breeze into functions (#7433)
2020-02-19 03:58:53 +03:00
Improve production image iteration speed (#9162) For a long time the way how entrypoint worked in ci scripts was wrong. The way it worked was convoluted and short of black magic. This did not allow to pass multiple test targets and required separate execute command scripts in Breeze. This is all now straightened out and both production and CI image are always using the right entrypoint by default and we can simply pass parameters to the image as usual without escaping strings. This also allowed to remove some breeze commands and change names of several flags in Breeze to make them more meaningful. Both CI and PROD image have now embedded scripts for log cleaning. History of image releases is added for 1.10.10-* alpha quality images.
2020-06-16 13:36:46 +03:00
# Prepare script for "run docker compose CI command"
Add packages to function names in bash (#10670) (#10696) Inspired by the Google Shell Guide where they mentioned separating package names with :: I realized that this was one of the missing pieces in the bash scripts of ours. While we already had packages (in libraries folders) it's been difficult to realise which function is where. With introducing packages - equal to the library file name we are *almost* at a level of a structured language - and it's easier to find the functions if you are looking for them. Way easier in fact. Part of #10576 (cherry picked from commit cc551ba793344800d2d396c13d7fd0c8eed97352) (cherry picked from commit 2bba276f0f06a5981bdd7e4f0e7e5ca2fe84f063)
2020-09-02 22:58:37 +03:00
breeze::prepare_command_file "${BUILD_CACHE_DIR}/${DOCKER_COMPOSE_RUN_SCRIPT_FOR_CI}" \
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
"\"\${@}\"" "${compose_ci_file}" "${AIRFLOW_CI_IMAGE}"
Add Production Docker image support (#7832)
2020-04-02 20:52:11 +03:00
Improve production image iteration speed (#9162) For a long time the way how entrypoint worked in ci scripts was wrong. The way it worked was convoluted and short of black magic. This did not allow to pass multiple test targets and required separate execute command scripts in Breeze. This is all now straightened out and both production and CI image are always using the right entrypoint by default and we can simply pass parameters to the image as usual without escaping strings. This also allowed to remove some breeze commands and change names of several flags in Breeze to make them more meaningful. Both CI and PROD image have now embedded scripts for log cleaning. History of image releases is added for 1.10.10-* alpha quality images.
2020-06-16 13:36:46 +03:00
# Prepare script for "run docker compose PROD command"
Add packages to function names in bash (#10670) (#10696) Inspired by the Google Shell Guide where they mentioned separating package names with :: I realized that this was one of the missing pieces in the bash scripts of ours. While we already had packages (in libraries folders) it's been difficult to realise which function is where. With introducing packages - equal to the library file name we are *almost* at a level of a structured language - and it's easier to find the functions if you are looking for them. Way easier in fact. Part of #10576 (cherry picked from commit cc551ba793344800d2d396c13d7fd0c8eed97352) (cherry picked from commit 2bba276f0f06a5981bdd7e4f0e7e5ca2fe84f063)
2020-09-02 22:58:37 +03:00
breeze::prepare_command_file "${BUILD_CACHE_DIR}/${DOCKER_COMPOSE_RUN_SCRIPT_FOR_PROD}" \
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
"\"\${@}\"" "${compose_prod_file}" "${AIRFLOW_PROD_IMAGE}"
[AIRFLOW-6820] split breeze into functions (#7433)
2020-02-19 03:58:53 +03:00
}
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
#######################################################################################################
#
Update github flags for Breeze (#10384) Part of #10368
2020-08-19 01:53:04 +03:00
# Prints detailed help for all commands and flags. Used to generate documentation added to BREEZE.rst
Add comments to breeze scripts (#8797)
2020-05-10 20:55:40 +03:00
# automatically.
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
#
Make breeeze-complete Google Shell Guide compatible (#10708) Also added unit tests for breeze-complete Part of #10576
2020-09-14 11:21:09 +03:00
# Used global variables:
# _breeze_all_commands
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
#
# Outputs:
# Prints detailed help for all commands to stdout.
#
#######################################################################################################
Add packages to function names in bash (#10670) (#10696) Inspired by the Google Shell Guide where they mentioned separating package names with :: I realized that this was one of the missing pieces in the bash scripts of ours. While we already had packages (in libraries folders) it's been difficult to realise which function is where. With introducing packages - equal to the library file name we are *almost* at a level of a structured language - and it's easier to find the functions if you are looking for them. Way easier in fact. Part of #10576 (cherry picked from commit cc551ba793344800d2d396c13d7fd0c8eed97352) (cherry picked from commit 2bba276f0f06a5981bdd7e4f0e7e5ca2fe84f063)
2020-09-02 22:58:37 +03:00
function breeze::do_help_all() {
[AIRFLOW-6838] Introduce real subcommands for Breeze (#7515) This change introduces sub-commands in breeze tool. It is much needed as we have many commands now and it was difficult to separate commands from flags. Also --help output was very long and unreadable. With this change help it is much easier to discover what breeze can do for you as well as navigate with it. Co-authored-by: Jarek Potiuk <jarek@potiuk.com> Co-authored-by: Kamil Breguła <mik-laj@users.noreply.github.com>
2020-02-25 00:31:50 +03:00
echo
Add packages to function names in bash (#10670) (#10696) Inspired by the Google Shell Guide where they mentioned separating package names with :: I realized that this was one of the missing pieces in the bash scripts of ours. While we already had packages (in libraries folders) it's been difficult to realise which function is where. With introducing packages - equal to the library file name we are *almost* at a level of a structured language - and it's easier to find the functions if you are looking for them. Way easier in fact. Part of #10576 (cherry picked from commit cc551ba793344800d2d396c13d7fd0c8eed97352) (cherry picked from commit 2bba276f0f06a5981bdd7e4f0e7e5ca2fe84f063)
2020-09-02 22:58:37 +03:00
breeze::print_line
breeze::usage
breeze::print_line
[AIRFLOW-6838] Introduce real subcommands for Breeze (#7515) This change introduces sub-commands in breeze tool. It is much needed as we have many commands now and it was difficult to separate commands from flags. Also --help output was very long and unreadable. With this change help it is much easier to discover what breeze can do for you as well as navigate with it. Co-authored-by: Jarek Potiuk <jarek@potiuk.com> Co-authored-by: Kamil Breguła <mik-laj@users.noreply.github.com>
2020-02-25 00:31:50 +03:00
echo
echo
echo "Detailed usage"
echo
Add packages to function names in bash (#10670) (#10696) Inspired by the Google Shell Guide where they mentioned separating package names with :: I realized that this was one of the missing pieces in the bash scripts of ours. While we already had packages (in libraries folders) it's been difficult to realise which function is where. With introducing packages - equal to the library file name we are *almost* at a level of a structured language - and it's easier to find the functions if you are looking for them. Way easier in fact. Part of #10576 (cherry picked from commit cc551ba793344800d2d396c13d7fd0c8eed97352) (cherry picked from commit 2bba276f0f06a5981bdd7e4f0e7e5ca2fe84f063)
2020-09-02 22:58:37 +03:00
breeze::print_line
[AIRFLOW-6838] Introduce real subcommands for Breeze (#7515) This change introduces sub-commands in breeze tool. It is much needed as we have many commands now and it was difficult to separate commands from flags. Also --help output was very long and unreadable. With this change help it is much easier to discover what breeze can do for you as well as navigate with it. Co-authored-by: Jarek Potiuk <jarek@potiuk.com> Co-authored-by: Kamil Breguła <mik-laj@users.noreply.github.com>
2020-02-25 00:31:50 +03:00
echo
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
local subcommand
Make breeeze-complete Google Shell Guide compatible (#10708) Also added unit tests for breeze-complete Part of #10576
2020-09-14 11:21:09 +03:00
# shellcheck disable=SC2154
for subcommand in ${_breeze_all_commands}; do
Add packages to function names in bash (#10670) (#10696) Inspired by the Google Shell Guide where they mentioned separating package names with :: I realized that this was one of the missing pieces in the bash scripts of ours. While we already had packages (in libraries folders) it's been difficult to realise which function is where. With introducing packages - equal to the library file name we are *almost* at a level of a structured language - and it's easier to find the functions if you are looking for them. Way easier in fact. Part of #10576 (cherry picked from commit cc551ba793344800d2d396c13d7fd0c8eed97352) (cherry picked from commit 2bba276f0f06a5981bdd7e4f0e7e5ca2fe84f063)
2020-09-02 22:58:37 +03:00
breeze::detailed_usage "${subcommand}"
breeze::print_line
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
echo
[AIRFLOW-6838] Introduce real subcommands for Breeze (#7515) This change introduces sub-commands in breeze tool. It is much needed as we have many commands now and it was difficult to separate commands from flags. Also --help output was very long and unreadable. With this change help it is much easier to discover what breeze can do for you as well as navigate with it. Co-authored-by: Jarek Potiuk <jarek@potiuk.com> Co-authored-by: Kamil Breguła <mik-laj@users.noreply.github.com>
2020-02-25 00:31:50 +03:00
done
echo
Add packages to function names in bash (#10670) (#10696) Inspired by the Google Shell Guide where they mentioned separating package names with :: I realized that this was one of the missing pieces in the bash scripts of ours. While we already had packages (in libraries folders) it's been difficult to realise which function is where. With introducing packages - equal to the library file name we are *almost* at a level of a structured language - and it's easier to find the functions if you are looking for them. Way easier in fact. Part of #10576 (cherry picked from commit cc551ba793344800d2d396c13d7fd0c8eed97352) (cherry picked from commit 2bba276f0f06a5981bdd7e4f0e7e5ca2fe84f063)
2020-09-02 22:58:37 +03:00
breeze::flags
[AIRFLOW-6838] Introduce real subcommands for Breeze (#7515) This change introduces sub-commands in breeze tool. It is much needed as we have many commands now and it was difficult to separate commands from flags. Also --help output was very long and unreadable. With this change help it is much easier to discover what breeze can do for you as well as navigate with it. Co-authored-by: Jarek Potiuk <jarek@potiuk.com> Co-authored-by: Kamil Breguła <mik-laj@users.noreply.github.com>
2020-02-25 00:31:50 +03:00
}
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
#######################################################################################################
#
Add comments to breeze scripts (#8797)
2020-05-10 20:55:40 +03:00
# Parses all arguments that can be passed to Breeze command - that includes command to run and flags.
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
#
Make breeeze-complete Google Shell Guide compatible (#10708) Also added unit tests for breeze-complete Part of #10576
2020-09-14 11:21:09 +03:00
# Used global variables:
# _breeze_getopt_short_options
# _breeze_getopt_long_options
# _breeze_allowed_integrations
#
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
# Updated global constants:
Make breeeze-complete Google Shell Guide compatible (#10708) Also added unit tests for breeze-complete Part of #10576
2020-09-14 11:21:09 +03:00
# By the end of this function, all the constants from `initialization::make_constants_read_only`
# function are set and they are set as read-only.
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
#
#######################################################################################################
Add packages to function names in bash (#10670) (#10696) Inspired by the Google Shell Guide where they mentioned separating package names with :: I realized that this was one of the missing pieces in the bash scripts of ours. While we already had packages (in libraries folders) it's been difficult to realise which function is where. With introducing packages - equal to the library file name we are *almost* at a level of a structured language - and it's easier to find the functions if you are looking for them. Way easier in fact. Part of #10576 (cherry picked from commit cc551ba793344800d2d396c13d7fd0c8eed97352) (cherry picked from commit 2bba276f0f06a5981bdd7e4f0e7e5ca2fe84f063)
2020-09-02 22:58:37 +03:00
function breeze::parse_arguments() {
[AIRFLOW-6841] Fixed unbounded variable on Mac (#7465)
2020-02-19 19:35:15 +03:00
set -u
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
local params
if ! params=$(getopt \
Make breeeze-complete Google Shell Guide compatible (#10708) Also added unit tests for breeze-complete Part of #10576
2020-09-14 11:21:09 +03:00
-o "${_breeze_getopt_short_options:=}" \
-l "${_breeze_getopt_long_options:=}" \
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
--name "$CMDNAME" -- "$@"); then
Fix `breeze flags` command. (#10766) This was missed in #10670
2020-09-06 14:32:32 +03:00
breeze::flags
[AIRFLOW-6820] split breeze into functions (#7433)
2020-02-19 03:58:53 +03:00
exit 1
fi
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
eval set -- "${params}"
unset params
[AIRFLOW-6820] split breeze into functions (#7433)
2020-02-19 03:58:53 +03:00
# Parse Flags.
# Please update short and long options in the breeze-complete script
# This way autocomplete will work out-of-the-box
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
while true; do
case "${1}" in
-h | --help)
run_help="true"
shift
;;
-p | --python)
export PYTHON_MAJOR_MINOR_VERSION="${2}"
echo "Python version: ${PYTHON_MAJOR_MINOR_VERSION}"
echo
shift 2
;;
-b | --backend)
export BACKEND="${2}"
echo "Backend: ${BACKEND}"
echo
shift 2
;;
-i | --integration)
Fix `breeze -i` error (#10887) It was erroring with "unbound" variable, expecting INTEGRATION to be a pre-defined variable.
2020-09-12 00:25:43 +03:00
local INTEGRATION=${2}
Add packages to function names in bash (#10670) (#10696) Inspired by the Google Shell Guide where they mentioned separating package names with :: I realized that this was one of the missing pieces in the bash scripts of ours. While we already had packages (in libraries folders) it's been difficult to realise which function is where. With introducing packages - equal to the library file name we are *almost* at a level of a structured language - and it's easier to find the functions if you are looking for them. Way easier in fact. Part of #10576 (cherry picked from commit cc551ba793344800d2d396c13d7fd0c8eed97352) (cherry picked from commit 2bba276f0f06a5981bdd7e4f0e7e5ca2fe84f063)
2020-09-02 22:58:37 +03:00
parameters::check_and_save_allowed_param "INTEGRATION" "integration" "--integration"
Fix `breeze -i` error (#10887) It was erroring with "unbound" variable, expecting INTEGRATION to be a pre-defined variable.
2020-09-12 00:25:43 +03:00
echo "Integration: ${INTEGRATION}"
if [[ ${INTEGRATION} == "all" ]]; then
Make breeeze-complete Google Shell Guide compatible (#10708) Also added unit tests for breeze-complete Part of #10576
2020-09-14 11:21:09 +03:00
# shellcheck disable=SC2154
for INTEGRATION in ${_breeze_allowed_integrations}; do
Fix `breeze -i` error (#10887) It was erroring with "unbound" variable, expecting INTEGRATION to be a pre-defined variable.
2020-09-12 00:25:43 +03:00
if [[ ${INTEGRATION} != "all" ]]; then
echo "${INTEGRATION}"
INTEGRATIONS+=("${INTEGRATION}")
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
fi
done
else
INTEGRATIONS+=("${INTEGRATION}")
fi
echo
shift 2
;;
-K | --kubernetes-mode)
export KUBERNETES_MODE="${2}"
echo "Kubernetes mode: ${KUBERNETES_MODE}"
echo
shift 2
;;
-V | --kubernetes-version)
export KUBERNETES_VERSION="${2}"
echo "Kubernetes version: ${KUBERNETES_VERSION}"
echo
shift 2
;;
Switches to Helm Chart for Kubernetes tests (#9468) The Kubernetes tests are now run using Helm chart rather than the custom templates we used to have. The Helm Chart uses locally build production image so the tests are testing not only Airflow but also Helm Chart and a Production image - all at the same time. Later on we will add more tests covering more functionalities of both Helm Chart and Production Image. This is the first step to get all of those bundle together and become testable. This change introduces also 'shell' sub-command for Breeze's kind-cluster command and EMBEDDED_DAGS build args for production image - both of them useful to run the Kubernetes tests more easily - without building two images and with an easy-to-iterate-over-tests shell command - which works without any other development environment. Co-authored-by: Jarek Potiuk <jarek@potiuk.com> Co-authored-by: Daniel Imberman <daniel@astronomer.io>
2020-07-01 15:50:30 +03:00
--kind-version)
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
export KIND_VERSION="${2}"
echo "Kind version: ${KIND_VERSION}"
echo
shift 2
;;
Switches to Helm Chart for Kubernetes tests (#9468) The Kubernetes tests are now run using Helm chart rather than the custom templates we used to have. The Helm Chart uses locally build production image so the tests are testing not only Airflow but also Helm Chart and a Production image - all at the same time. Later on we will add more tests covering more functionalities of both Helm Chart and Production Image. This is the first step to get all of those bundle together and become testable. This change introduces also 'shell' sub-command for Breeze's kind-cluster command and EMBEDDED_DAGS build args for production image - both of them useful to run the Kubernetes tests more easily - without building two images and with an easy-to-iterate-over-tests shell command - which works without any other development environment. Co-authored-by: Jarek Potiuk <jarek@potiuk.com> Co-authored-by: Daniel Imberman <daniel@astronomer.io>
2020-07-01 15:50:30 +03:00
--helm-version)
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
export HELM_VERSION="${2}"
echo "Helm version: ${HELM_VERSION}"
echo
shift 2
;;
[AIRFLOW-7058] Add support for different DB versions (#7717)
2020-03-15 01:57:47 +03:00
--postgres-version)
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
export POSTGRES_VERSION="${2}"
echo "Postgres version: ${POSTGRES_VERSION}"
echo
shift 2
;;
[AIRFLOW-7058] Add support for different DB versions (#7717)
2020-03-15 01:57:47 +03:00
--mysql-version)
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
export MYSQL_VERSION="${2}"
echo "MySQL version: ${MYSQL_VERSION}"
echo
shift 2
;;
-l | --skip-mounting-local-sources)
MOUNT_LOCAL_SOURCES="false"
echo "Mount local sources: ${MOUNT_LOCAL_SOURCES}"
echo
shift
;;
-a | --install-airflow-version)
INSTALL_AIRFLOW_VERSION="${2}"
# Reference is mutually exclusive with version
INSTALL_AIRFLOW_REFERENCE=""
echo "Installs version of Airflow: ${INSTALL_AIRFLOW_VERSION}"
echo
shift 2
;;
-t | --install-airflow-reference)
INSTALL_AIRFLOW_REFERENCE="${2}"
# Reference is mutually exclusive with version
INSTALL_AIRFLOW_VERSION=""
echo "Installs Airflow from reference: ${INSTALL_AIRFLOW_REFERENCE}"
echo
shift 2
;;
-d | --db-reset)
echo "Resetting the DB!"
echo
export DB_RESET="true"
Make the usage of bash 'shift' consistent across Breeze (#10907) If an argument is not provided to `shift` it by default uses `shift 1`. The commands "shift 1" and "shift" (with no argument) do the same thing. This PR makes the usage consistent as there were occurences of both
2020-09-13 17:28:11 +03:00
shift
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
;;
-v | --verbose)
export VERBOSE="true"
echo "Verbose output"
echo
shift
;;
-y | --assume-yes)
export FORCE_ANSWER_TO_QUESTIONS="yes"
echo "Assuming 'yes' answer to all questions."
echo
shift
;;
-n | --assume-no)
export FORCE_ANSWER_TO_QUESTIONS="no"
echo "Assuming 'no' answer to all questions."
echo
shift
;;
-q | --assume-quit)
export FORCE_ANSWER_TO_QUESTIONS="quit"
echo "Assuming 'quit' answer to all questions."
echo
shift
;;
-F | --force-build-images)
echo "Force build images"
echo
export FORCE_BUILD_IMAGES="true"
# if you want to force build an image - assume you want to build it :)
export FORCE_ANSWER_TO_QUESTIONS="yes"
shift
;;
-C | --force-clean-images)
echo "Clean build of images without cache"
echo
export DOCKER_CACHE="no-cache"
# if not set here, docker cached is determined later, depending on type of image to be build
readonly DOCKER_CACHE
export FORCE_BUILD_IMAGES="true"
shift
;;
-r | --skip-rebuild-check)
echo "Skips checking image for rebuilds"
echo
export CHECK_IMAGE_FOR_REBUILD="false"
export SKIP_BUILDING_PROD_IMAGE="true"
shift
;;
-L | --build-cache-local)
echo "Use local cache to build images"
echo
export DOCKER_CACHE="local"
# if not set here, docker cached is determined later, depending on type of image to be build
readonly DOCKER_CACHE
shift
;;
-U | --build-cache-pulled)
echo "Use pulled cache to build images"
echo
export DOCKER_CACHE="pulled"
# if not set here, docker cached is determined later, depending on type of image to be build
readonly DOCKER_CACHE
shift
;;
-X | --build-cache-disabled)
echo "Use disabled cache to build images"
echo
export DOCKER_CACHE="disabled"
readonly DOCKER_CACHE
# if not set here, docker cached is determined later, depending on type of image to be build
shift
;;
Optionally disables PIP cache from GitHub during the build (#11173) This is first step of implementing the corporate-environment friendly way of building images, where in the corporate environment, this might not be possible to install the packages using the GitHub cache initially. Part of #11171
2020-09-27 19:00:03 +03:00
-B | --disable-pip-cache)
echo "Disable PIP cache during build"
echo
export AIRFLOW_PRE_CACHED_PIP_PACKAGES="false"
shift
;;
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
-P | --force-pull-images)
echo "Force pulling images before build. Uses pulled images as cache."
echo
export FORCE_PULL_IMAGES="true"
export FORCE_BUILD_IMAGES="true"
# if you want to force build an image - assume you want to build it :)
export FORCE_ANSWER_TO_QUESTIONS="yes"
shift
;;
-I | --production-image)
export PRODUCTION_IMAGE="true"
export SQLITE_URL=
echo
echo "*************** PRODUCTION IMAGE *************************"
echo
shift
;;
-E | --extras)
export AIRFLOW_EXTRAS="${2}"
echo "Extras : ${AIRFLOW_EXTRAS}"
shift 2
;;
Additional python extras and deps can be set in breeze (#9035) Closes #8604 Closes #8866
2020-05-27 18:09:11 +03:00
--additional-extras)
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
export ADDITIONAL_AIRFLOW_EXTRAS="${2}"
echo "Additional extras : ${ADDITIONAL_AIRFLOW_EXTRAS}"
shift 2
;;
Additional python extras and deps can be set in breeze (#9035) Closes #8604 Closes #8866
2020-05-27 18:09:11 +03:00
--additional-python-deps)
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
export ADDITIONAL_PYTHON_DEPS="${2}"
echo "Additional python dependencies: ${ADDITIONAL_PYTHON_DEPS}"
shift 2
;;
Additional apt dependencies options in breeze (#9231)
2020-06-11 19:53:26 +03:00
--additional-dev-deps)
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
export ADDITIONAL_DEV_DEPS="${2}"
echo "Additional apt dev dependencies: ${ADDITIONAL_DEV_DEPS}"
shift 2
;;
Additional apt dependencies options in breeze (#9231)
2020-06-11 19:53:26 +03:00
--additional-runtime-deps)
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
export ADDITIONAL_RUNTIME_DEPS="${2}"
echo "Additional apt runtime dependencies: ${ADDITIONAL_RUNTIME_DEPS}"
shift 2
;;
Conditional MySQL Client installation (#11174) This is the second step of making the Production Docker Image more corporate-environment friendly, by making MySQL client installation optional. Instaling MySQL Client on Debian requires to reach out to oracle deb repositories which might not be approved by security teams when you build the images. Also not everyone needs MySQL client or might want to install their own MySQL client or MariaDB client - from their own repositories. This change makes the installation step separated out to script (with prod/dev installation option). The prod/dev separation is needed because MySQL needs to be installed with dev libraries in the "Build" segment of the image (requiring build essentials etc.) but in "Final" segment of the image only runtime libraries are needed. Part of #11171 Depends on #11173.
2020-09-27 19:56:58 +03:00
--disable-mysql-client-installation)
export INSTALL_MYSQL_CLIENT="false"
echo "Install MySQL client: ${INSTALL_MYSQL_CLIENT}"
shift
;;
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
-D | --dockerhub-user)
export DOCKERHUB_USER="${2}"
echo "Dockerhub user ${DOCKERHUB_USER}"
echo
shift 2
;;
-R | --dockerhub-repo)
export DOCKERHUB_REPO="${2}"
echo "Dockerhub repo ${DOCKERHUB_REPO}"
echo
shift 2
;;
-f | --forward-credentials)
echo "Forwarding credentials. Be careful as your credentials ar available in the container!"
echo
export FORWARD_CREDENTIALS="true"
Make the usage of bash 'shift' consistent across Breeze (#10907) If an argument is not provided to `shift` it by default uses `shift 1`. The commands "shift 1" and "shift" (with no argument) do the same thing. This PR makes the usage consistent as there were occurences of both
2020-09-13 17:28:11 +03:00
shift
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
;;
-c | --github-registry)
echo
echo "Use github registry"
echo
export USE_GITHUB_REGISTRY="true"
shift
;;
-g | --github-repository)
echo
echo "GitHub repository: ${2}"
echo
echo "Using github registry."
echo
export GITHUB_REPOSITORY="${2}"
export USE_GITHUB_REGISTRY="true"
shift 2
;;
-s | --github-image-id)
echo
Remove redundant curly brace from breeze echo message (#11012) Before: ``` ❯ ./breeze --github-image-id 260274893 GitHub image id: 260274893} ``` After: ``` ❯ ./breeze --github-image-id 260274893 GitHub image id: 260274893 ```
2020-09-18 16:51:16 +03:00
echo "GitHub image id: ${2}"
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
echo
echo "Force pulling the image, using github registry and skip mounting local sources."
echo "This is in order to get the exact same version as used in CI environment for SHA/RUN_ID!."
echo
export FORCE_PULL_IMAGES="true"
export USE_GITHUB_REGISTRY="true"
export GITHUB_REGISTRY_PULL_IMAGE_TAG="${2}"
export GITHUB_REGISTRY_PUSH_IMAGE_TAG="${2}"
export CHECK_IMAGE_FOR_REBUILD="false"
export SKIP_BUILDING_PROD_IMAGE="true"
export MOUNT_LOCAL_SOURCES="false"
export SKIP_CHECK_REMOTE_IMAGE="true"
shift 2
;;
Starting breeze will run an init script after the environment is setup (#11029) Added the possibility to run an init script
2020-09-21 13:58:30 +03:00
--init-script)
export INIT_SCRIPT_FILE="${2}"
echo "The initialization file is in ${INIT_SCRIPT_FILE}"
echo
shift 2
;;
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
-S | --version-suffix-for-pypi)
if [[ -n ${VERSION_SUFFIX_FOR_SVN=} ]]; then
echo
echo "You can only set one version suffix - either for PyPI or for SVN"
echo
exit 1
fi
export VERSION_SUFFIX_FOR_PYPI="${2}"
echo "Version suffix for PyPI ${VERSION_SUFFIX_FOR_PYPI}"
echo
shift 2
;;
-N | --version-suffix-for-svn)
if [[ -n ${VERSION_SUFFIX_FOR_PYPI=} ]]; then
echo
echo "You can only set one version suffix - either for PyPI or for SVN"
echo
exit 1
fi
export VERSION_SUFFIX_FOR_SVN="${2}"
echo "Version suffix for SVN ${VERSION_SUFFIX_FOR_SVN}"
echo
shift 2
;;
New Breeze command start-airflow, it replaces the previous flag (#11157)
2020-09-27 19:31:50 +03:00
--load-example-dags)
export LOAD_EXAMPLES="true"
echo "Include Airflow sample dags"
echo
shift
;;
--load-default-connections)
export LOAD_DEFAULT_CONNECTIONS="true"
echo "Include Airflow default connections"
echo
shift
;;
[AIRFLOW-6838] Introduce real subcommands for Breeze (#7515) This change introduces sub-commands in breeze tool. It is much needed as we have many commands now and it was difficult to separate commands from flags. Also --help output was very long and unreadable. With this change help it is much easier to discover what breeze can do for you as well as navigate with it. Co-authored-by: Jarek Potiuk <jarek@potiuk.com> Co-authored-by: Kamil Breguła <mik-laj@users.noreply.github.com>
2020-02-25 00:31:50 +03:00
--)
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
shift
break
;;
[AIRFLOW-6838] Introduce real subcommands for Breeze (#7515) This change introduces sub-commands in breeze tool. It is much needed as we have many commands now and it was difficult to separate commands from flags. Also --help output was very long and unreadable. With this change help it is much easier to discover what breeze can do for you as well as navigate with it. Co-authored-by: Jarek Potiuk <jarek@potiuk.com> Co-authored-by: Kamil Breguła <mik-laj@users.noreply.github.com>
2020-02-25 00:31:50 +03:00
*)
Fix `breeze flags` command. (#10766) This was missed in #10670
2020-09-06 14:32:32 +03:00
breeze::flags
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
echo >&2
echo >&2 "ERROR: Unknown flag ${1}"
echo >&2
exit 1
;;
esac
[AIRFLOW-6838] Introduce real subcommands for Breeze (#7515) This change introduces sub-commands in breeze tool. It is much needed as we have many commands now and it was difficult to separate commands from flags. Also --help output was very long and unreadable. With this change help it is much easier to discover what breeze can do for you as well as navigate with it. Co-authored-by: Jarek Potiuk <jarek@potiuk.com> Co-authored-by: Kamil Breguła <mik-laj@users.noreply.github.com>
2020-02-25 00:31:50 +03:00
done
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
local last_subcommand=""
Help for breeze commands contain relevant flags. (#8261) * Help for breeze commands contain relevant flags. * fixup! Help for breeze commands contain relevant flags. * fixup! fixup! Help for breeze commands contain relevant flags.
2020-04-13 09:51:33 +03:00
# Parse commands
[AIRFLOW-6838] Introduce real subcommands for Breeze (#7515) This change introduces sub-commands in breeze tool. It is much needed as we have many commands now and it was difficult to separate commands from flags. Also --help output was very long and unreadable. With this change help it is much easier to discover what breeze can do for you as well as navigate with it. Co-authored-by: Jarek Potiuk <jarek@potiuk.com> Co-authored-by: Kamil Breguła <mik-laj@users.noreply.github.com>
2020-02-25 00:31:50 +03:00
if [[ "$#" -ne 0 ]]; then
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
case "${1}" in
[AIRFLOW-6838] Introduce real subcommands for Breeze (#7515) This change introduces sub-commands in breeze tool. It is much needed as we have many commands now and it was difficult to separate commands from flags. Also --help output was very long and unreadable. With this change help it is much easier to discover what breeze can do for you as well as navigate with it. Co-authored-by: Jarek Potiuk <jarek@potiuk.com> Co-authored-by: Kamil Breguła <mik-laj@users.noreply.github.com>
2020-02-25 00:31:50 +03:00
shell)
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
last_subcommand="${1}"
shift
;;
[AIRFLOW-7005] Added exec command to Breeze (#7649)
2020-03-07 22:18:51 +03:00
exec)
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
last_subcommand="${1}"
command_to_run="run_exec"
shift
;;
[AIRFLOW-6838] Introduce real subcommands for Breeze (#7515) This change introduces sub-commands in breeze tool. It is much needed as we have many commands now and it was difficult to separate commands from flags. Also --help output was very long and unreadable. With this change help it is much easier to discover what breeze can do for you as well as navigate with it. Co-authored-by: Jarek Potiuk <jarek@potiuk.com> Co-authored-by: Kamil Breguła <mik-laj@users.noreply.github.com>
2020-02-25 00:31:50 +03:00
build-docs)
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
last_subcommand="${1}"
command_to_run="build_docs"
Make the usage of bash 'shift' consistent across Breeze (#10907) If an argument is not provided to `shift` it by default uses `shift 1`. The commands "shift 1" and "shift" (with no argument) do the same thing. This PR makes the usage consistent as there were occurences of both
2020-09-13 17:28:11 +03:00
shift
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
;;
Add Production Docker image support (#7832)
2020-04-02 20:52:11 +03:00
build-image)
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
last_subcommand="${1}"
command_to_run="build_image"
# if you want to build an image - assume you want to build it :)
export FORCE_ANSWER_TO_QUESTIONS="yes"
# and assume you want to build it no matter if it is needed
export FORCE_BUILD_IMAGES="true"
echo "Build image"
echo
shift
;;
Add Production Docker image support (#7832)
2020-04-02 20:52:11 +03:00
cleanup-image)
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
last_subcommand="${1}"
echo "Cleanup the image"
echo
command_to_run="cleanup_image"
Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals.
2020-08-31 14:24:53 +03:00
shift
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
;;
docker-compose)
last_subcommand="${1}"
if [[ $# -lt 2 ]]; then
echo "You should specify docker compose command to run"
shift
run_help="true"
else
docker_compose_command="${2}"
shift 2
fi
command_to_run="run_docker_compose"
;;
Revert recent breeze changes (#10651 & #10670) (#10694) * Revert "Add packages to function names in bash (#10670)" This reverts commit cc551ba793344800d2d396c13d7fd0c8eed97352. * Revert "Implement Google Shell Conventions for breeze script … (#10651)" This reverts commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2.
2020-09-02 19:27:36 +03:00
generate-constraints)
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
last_subcommand="${1}"
command_to_run="perform_generate_constraints"
export FORCE_ANSWER_TO_QUESTIONS="yes"
export FORCE_BUILD_IMAGES="true"
export UPGRADE_TO_LATEST_CONSTRAINTS="true"
shift
;;
Added automated release notes generation for backport operators (#8807) We have now mechanism to keep release notes updated for the backport operators in an automated way. It really nicely generates all the necessary information: * summary of requirements for each backport package * list of dependencies (including extras to install them) when package depends on other providers packages * table of new hooks/operators/sensors/protocols/secrets * table of moved hooks/operators/sensors/protocols/secrets with information where they were moved from * changelog of all the changes to the provider package (this will be automatically updated with incremental changelog whenever we decide to release separate packages. The system is fully automated - we will be able to produce release notes automatically (per-package) whenever we decide to release new version of the package in the future.
2020-05-15 20:00:15 +03:00
prepare-backport-packages)
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
last_subcommand="${1}"
command_to_run="perform_prepare_backport_packages"
shift
;;
Release candidate 2 for backport packages 2020.05.20 (#8898) Release candidate 2 for backport packages 2020.05.20
2020-05-19 15:17:22 +03:00
prepare-backport-readme)
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
last_subcommand="${1}"
command_to_run="perform_prepare_backport_readme"
shift
;;
You can push with Breeze as separate command and to cache (#8976) Breeze had --push-images switch to also push images to repo but it was often needed to build and push images separately. We have now a possibility to push an already built image with separate push-image command instead and also you can choose to push to cache registry in GitHub rather than to DockerHub with --registry-cache switch.
2020-06-02 11:24:47 +03:00
push-image)
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
last_subcommand="${1}"
command_to_run="perform_push_image"
export SKIP_CHECK_REMOTE_IMAGE="true"
shift
;;
Revert recent breeze changes (#10651 & #10670) (#10694) * Revert "Add packages to function names in bash (#10670)" This reverts commit cc551ba793344800d2d396c13d7fd0c8eed97352. * Revert "Implement Google Shell Conventions for breeze script … (#10651)" This reverts commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2.
2020-09-02 19:27:36 +03:00
initialize-local-virtualenv)
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
last_subcommand="${1}"
echo "Initializing local virtualenv"
echo
command_to_run="perform_initialize_local_virtualenv"
shift
;;
Revert recent breeze changes (#10651 & #10670) (#10694) * Revert "Add packages to function names in bash (#10670)" This reverts commit cc551ba793344800d2d396c13d7fd0c8eed97352. * Revert "Implement Google Shell Conventions for breeze script … (#10651)" This reverts commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2.
2020-09-02 19:27:36 +03:00
kind-cluster)
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
last_subcommand="${1}"
# Switch to production image for all kind operations
export PRODUCTION_IMAGE="true"
command_to_run="manage_kind_cluster"
export KIND_CLUSTER_OPERATION="${2:-}"
if [[ -n ${KIND_CLUSTER_OPERATION=} ]]; then
shift 2
else
shift
fi
;;
[AIRFLOW-6838] Introduce real subcommands for Breeze (#7515) This change introduces sub-commands in breeze tool. It is much needed as we have many commands now and it was difficult to separate commands from flags. Also --help output was very long and unreadable. With this change help it is much easier to discover what breeze can do for you as well as navigate with it. Co-authored-by: Jarek Potiuk <jarek@potiuk.com> Co-authored-by: Kamil Breguła <mik-laj@users.noreply.github.com>
2020-02-25 00:31:50 +03:00
setup-autocomplete)
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
last_subcommand="${1}"
echo "Setting up autocomplete"
echo
command_to_run="perform_setup_autocomplete"
shift
;;
static-check)
last_subcommand="${1}"
command_to_run="perform_static_checks"
if [[ "$#" -lt 2 ]]; then
if [[ ${run_help} != "true" ]]; then
echo "You should specify static check that you would like to run or 'all' to run all checks."
echo
echo "One of :"
echo
Make breeeze-complete Google Shell Guide compatible (#10708) Also added unit tests for breeze-complete Part of #10576
2020-09-14 11:21:09 +03:00
echo "${_breeze_allowed_static_checks:=}"
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
echo
echo "For example:"
echo
echo "${CMDNAME} static-check mypy"
echo
exit 1
else
shift
fi
Useful help information in test-target and docker-compose commands (#8796) There was no useful information printed in test-target and docker-compose commands. It's fixed now
2020-05-11 12:58:25 +03:00
else
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
export PYTHON_MAJOR_MINOR_VERSION=${DEFAULT_PYTHON_MAJOR_MINOR_VERSION}
static_check="${2:-}"
EXTRA_STATIC_CHECK_OPTIONS+=("--show-diff-on-failure")
shift 2
Useful help information in test-target and docker-compose commands (#8796) There was no useful information printed in test-target and docker-compose commands. It's fixed now
2020-05-11 12:58:25 +03:00
fi
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
;;
New Breeze command start-airflow, it replaces the previous flag (#11157)
2020-09-27 19:31:50 +03:00
start-airflow)
last_subcommand="${1}"
export START_AIRFLOW="true"
shift
;;
[AIRFLOW-6972] Shorter frequently used commands in Breeze (#7608)
2020-03-04 03:21:11 +03:00
stop)
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
last_subcommand="${1}"
command_to_run="run_docker_compose"
docker_compose_command="down"
EXTRA_DC_OPTIONS+=("--remove-orphans")
shift
;;
[AIRFLOW-6972] Shorter frequently used commands in Breeze (#7608)
2020-03-04 03:21:11 +03:00
restart)
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
last_subcommand="${1}"
command_to_run="run_docker_compose"
docker_compose_command="down"
EXTRA_DC_OPTIONS+=("--remove-orphans")
second_command_to_run="enter_breeze"
echo "Restarts the environment. Includes emptying the databases."
Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals.
2020-08-31 14:24:53 +03:00
shift
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
;;
tests)
last_subcommand="${1}"
if [[ $# -lt 2 ]]; then
run_help="true"
else
shift
fi
command_to_run="run_tests"
;;
Revert recent breeze changes (#10651 & #10670) (#10694) * Revert "Add packages to function names in bash (#10670)" This reverts commit cc551ba793344800d2d396c13d7fd0c8eed97352. * Revert "Implement Google Shell Conventions for breeze script … (#10651)" This reverts commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2.
2020-09-02 19:27:36 +03:00
toggle-suppress-cheatsheet)
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
last_subcommand="${1}"
if [[ -f "${SUPPRESS_CHEATSHEET_FILE}" ]]; then
rm -f "${SUPPRESS_CHEATSHEET_FILE}"
else
touch "${SUPPRESS_CHEATSHEET_FILE}"
fi
echo "Toggle suppress cheatsheet"
echo
shift
command_to_run="toggle_suppress_cheatsheet"
;;
[AIRFLOW-6838] Introduce real subcommands for Breeze (#7515) This change introduces sub-commands in breeze tool. It is much needed as we have many commands now and it was difficult to separate commands from flags. Also --help output was very long and unreadable. With this change help it is much easier to discover what breeze can do for you as well as navigate with it. Co-authored-by: Jarek Potiuk <jarek@potiuk.com> Co-authored-by: Kamil Breguła <mik-laj@users.noreply.github.com>
2020-02-25 00:31:50 +03:00
toggle-suppress-asciiart)
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
last_subcommand="${1}"
if [[ -f "${SUPPRESS_ASCIIART_FILE}" ]]; then
rm -f "${SUPPRESS_ASCIIART_FILE}"
else
touch "${SUPPRESS_ASCIIART_FILE}"
fi
echo "Toggle suppress asciiart"
echo
shift
command_to_run="toggle_suppress_asciiart"
;;
[AIRFLOW-6838] Introduce real subcommands for Breeze (#7515) This change introduces sub-commands in breeze tool. It is much needed as we have many commands now and it was difficult to separate commands from flags. Also --help output was very long and unreadable. With this change help it is much easier to discover what breeze can do for you as well as navigate with it. Co-authored-by: Jarek Potiuk <jarek@potiuk.com> Co-authored-by: Kamil Breguła <mik-laj@users.noreply.github.com>
2020-02-25 00:31:50 +03:00
flags)
Fix `breeze flags` command. (#10766) This was missed in #10670
2020-09-06 14:32:32 +03:00
breeze::flags
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
exit 0
;;
[AIRFLOW-6838] Introduce real subcommands for Breeze (#7515) This change introduces sub-commands in breeze tool. It is much needed as we have many commands now and it was difficult to separate commands from flags. Also --help output was very long and unreadable. With this change help it is much easier to discover what breeze can do for you as well as navigate with it. Co-authored-by: Jarek Potiuk <jarek@potiuk.com> Co-authored-by: Kamil Breguła <mik-laj@users.noreply.github.com>
2020-02-25 00:31:50 +03:00
help)
Add packages to function names in bash (#10670) (#10696) Inspired by the Google Shell Guide where they mentioned separating package names with :: I realized that this was one of the missing pieces in the bash scripts of ours. While we already had packages (in libraries folders) it's been difficult to realise which function is where. With introducing packages - equal to the library file name we are *almost* at a level of a structured language - and it's easier to find the functions if you are looking for them. Way easier in fact. Part of #10576 (cherry picked from commit cc551ba793344800d2d396c13d7fd0c8eed97352) (cherry picked from commit 2bba276f0f06a5981bdd7e4f0e7e5ca2fe84f063)
2020-09-02 22:58:37 +03:00
breeze::usage
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
exit 0
;;
[AIRFLOW-6838] Introduce real subcommands for Breeze (#7515) This change introduces sub-commands in breeze tool. It is much needed as we have many commands now and it was difficult to separate commands from flags. Also --help output was very long and unreadable. With this change help it is much easier to discover what breeze can do for you as well as navigate with it. Co-authored-by: Jarek Potiuk <jarek@potiuk.com> Co-authored-by: Kamil Breguła <mik-laj@users.noreply.github.com>
2020-02-25 00:31:50 +03:00
help-all)
Add packages to function names in bash (#10670) (#10696) Inspired by the Google Shell Guide where they mentioned separating package names with :: I realized that this was one of the missing pieces in the bash scripts of ours. While we already had packages (in libraries folders) it's been difficult to realise which function is where. With introducing packages - equal to the library file name we are *almost* at a level of a structured language - and it's easier to find the functions if you are looking for them. Way easier in fact. Part of #10576 (cherry picked from commit cc551ba793344800d2d396c13d7fd0c8eed97352) (cherry picked from commit 2bba276f0f06a5981bdd7e4f0e7e5ca2fe84f063)
2020-09-02 22:58:37 +03:00
breeze::do_help_all
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
exit 0
;;
[AIRFLOW-6820] split breeze into functions (#7433)
2020-02-19 03:58:53 +03:00
*)
Add packages to function names in bash (#10670) (#10696) Inspired by the Google Shell Guide where they mentioned separating package names with :: I realized that this was one of the missing pieces in the bash scripts of ours. While we already had packages (in libraries folders) it's been difficult to realise which function is where. With introducing packages - equal to the library file name we are *almost* at a level of a structured language - and it's easier to find the functions if you are looking for them. Way easier in fact. Part of #10576 (cherry picked from commit cc551ba793344800d2d396c13d7fd0c8eed97352) (cherry picked from commit 2bba276f0f06a5981bdd7e4f0e7e5ca2fe84f063)
2020-09-02 22:58:37 +03:00
breeze::usage
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
echo >&2
echo >&2 "ERROR: Unknown command ${1}"
echo >&2
exit 1
;;
esac
[AIRFLOW-6838] Introduce real subcommands for Breeze (#7515) This change introduces sub-commands in breeze tool. It is much needed as we have many commands now and it was difficult to separate commands from flags. Also --help output was very long and unreadable. With this change help it is much easier to discover what breeze can do for you as well as navigate with it. Co-authored-by: Jarek Potiuk <jarek@potiuk.com> Co-authored-by: Kamil Breguła <mik-laj@users.noreply.github.com>
2020-02-25 00:31:50 +03:00
else
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
:
# By default, start interactive terminal
[AIRFLOW-6838] Introduce real subcommands for Breeze (#7515) This change introduces sub-commands in breeze tool. It is much needed as we have many commands now and it was difficult to separate commands from flags. Also --help output was very long and unreadable. With this change help it is much easier to discover what breeze can do for you as well as navigate with it. Co-authored-by: Jarek Potiuk <jarek@potiuk.com> Co-authored-by: Kamil Breguła <mik-laj@users.noreply.github.com>
2020-02-25 00:31:50 +03:00
fi
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
if [[ ${run_help} == "true" ]]; then
if [[ ${last_subcommand} == "" ]]; then
Add packages to function names in bash (#10670) (#10696) Inspired by the Google Shell Guide where they mentioned separating package names with :: I realized that this was one of the missing pieces in the bash scripts of ours. While we already had packages (in libraries folders) it's been difficult to realise which function is where. With introducing packages - equal to the library file name we are *almost* at a level of a structured language - and it's easier to find the functions if you are looking for them. Way easier in fact. Part of #10576 (cherry picked from commit cc551ba793344800d2d396c13d7fd0c8eed97352) (cherry picked from commit 2bba276f0f06a5981bdd7e4f0e7e5ca2fe84f063)
2020-09-02 22:58:37 +03:00
breeze::usage
breeze::flag_footer
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
else
Add packages to function names in bash (#10670) (#10696) Inspired by the Google Shell Guide where they mentioned separating package names with :: I realized that this was one of the missing pieces in the bash scripts of ours. While we already had packages (in libraries folders) it's been difficult to realise which function is where. With introducing packages - equal to the library file name we are *almost* at a level of a structured language - and it's easier to find the functions if you are looking for them. Way easier in fact. Part of #10576 (cherry picked from commit cc551ba793344800d2d396c13d7fd0c8eed97352) (cherry picked from commit 2bba276f0f06a5981bdd7e4f0e7e5ca2fe84f063)
2020-09-02 22:58:37 +03:00
breeze::detailed_usage "${last_subcommand}"
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
fi
exit 0
[AIRFLOW-6838] Introduce real subcommands for Breeze (#7515) This change introduces sub-commands in breeze tool. It is much needed as we have many commands now and it was difficult to separate commands from flags. Also --help output was very long and unreadable. With this change help it is much easier to discover what breeze can do for you as well as navigate with it. Co-authored-by: Jarek Potiuk <jarek@potiuk.com> Co-authored-by: Kamil Breguła <mik-laj@users.noreply.github.com>
2020-02-25 00:31:50 +03:00
fi
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
# EXTRA_DC_OPTIONS is only used by Breeze. It's value is set here as well.
readonly EXTRA_DC_OPTIONS
# Also Remaining args are set here and set as read-only - no more changes to it.
[AIRFLOW-6820] split breeze into functions (#7433)
2020-02-19 03:58:53 +03:00
REMAINING_ARGS+=("$@")
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
export REMAINING_ARGS
readonly REMAINING_ARGS
[AIRFLOW-6820] split breeze into functions (#7433)
2020-02-19 03:58:53 +03:00
}
[AIRFLOW-3611] Simplified development environment (#4932)
2019-08-27 21:39:36 +03:00
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
#######################################################################################################
#
Add comments to breeze scripts (#8797)
2020-05-10 20:55:40 +03:00
# Prepares nicely formatted versions of list of allowed and default values defined in Breeze.
# It is used in help command to print the lists in a readable format and fold the lists
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
# so that they fit the screen indented_screen_width.
#
Make breeeze-complete Google Shell Guide compatible (#10708) Also added unit tests for breeze-complete Part of #10576
2020-09-14 11:21:09 +03:00
# Used global variables:
# _breeze_allowed_*
#
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
# Updated global constants:
# FORMATTED_* constant variables that can be used in Breeze Help output
#
#######################################################################################################
Add packages to function names in bash (#10670) (#10696) Inspired by the Google Shell Guide where they mentioned separating package names with :: I realized that this was one of the missing pieces in the bash scripts of ours. While we already had packages (in libraries folders) it's been difficult to realise which function is where. With introducing packages - equal to the library file name we are *almost* at a level of a structured language - and it's easier to find the functions if you are looking for them. Way easier in fact. Part of #10576 (cherry picked from commit cc551ba793344800d2d396c13d7fd0c8eed97352) (cherry picked from commit 2bba276f0f06a5981bdd7e4f0e7e5ca2fe84f063)
2020-09-02 22:58:37 +03:00
function breeze::prepare_formatted_versions() {
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
local indent=15
local list_prefix
list_prefix=$(printf "%-${indent}s" " ")
local indented_screen_width=$((SCREEN_WIDTH - indent))
Make breeeze-complete Google Shell Guide compatible (#10708) Also added unit tests for breeze-complete Part of #10576
2020-09-14 11:21:09 +03:00
FORMATTED_PYTHON_MAJOR_MINOR_VERSIONS=$(echo "${_breeze_allowed_python_major_minor_versions=""}" | tr '\n' ' ' |
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
fold -w "${indented_screen_width}" -s | sed "s/^/${list_prefix}/")
readonly FORMATTED_PYTHON_MAJOR_MINOR_VERSIONS
Make breeeze-complete Google Shell Guide compatible (#10708) Also added unit tests for breeze-complete Part of #10576
2020-09-14 11:21:09 +03:00
FORMATTED_BACKENDS=$(echo "${_breeze_allowed_backends=""}" | tr '\n' ' ' |
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
fold -w "${indented_screen_width}" -s | sed "s/^/${list_prefix}/")
readonly FORMATTED_BACKENDS
Make breeeze-complete Google Shell Guide compatible (#10708) Also added unit tests for breeze-complete Part of #10576
2020-09-14 11:21:09 +03:00
FORMATTED_STATIC_CHECKS=$(echo "${_breeze_allowed_static_checks=""}" | tr '\n' ' ' |
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
fold -w "${indented_screen_width}" -s | sed "s/^/${list_prefix}/")
readonly FORMATTED_STATIC_CHECKS
Make breeeze-complete Google Shell Guide compatible (#10708) Also added unit tests for breeze-complete Part of #10576
2020-09-14 11:21:09 +03:00
FORMATTED_INTEGRATIONS=$(echo "${_breeze_allowed_integrations=""}" | tr '\n' ' ' |
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
fold -w "${indented_screen_width}" -s | sed "s/^/${list_prefix}/")
readonly FORMATTED_INTEGRATIONS
Make breeeze-complete Google Shell Guide compatible (#10708) Also added unit tests for breeze-complete Part of #10576
2020-09-14 11:21:09 +03:00
FORMATTED_KUBERNETES_MODES=$(echo "${_breeze_allowed_kubernetes_modes=""}" | tr '\n' ' ' |
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
fold -w "${indented_screen_width}" -s | sed "s/^/${list_prefix}/")
readonly FORMATTED_KUBERNETES_MODES
Make breeeze-complete Google Shell Guide compatible (#10708) Also added unit tests for breeze-complete Part of #10576
2020-09-14 11:21:09 +03:00
FORMATTED_KUBERNETES_VERSIONS=$(echo "${_breeze_allowed_kubernetes_versions=""}" | tr '\n' ' ' |
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
fold -w "${indented_screen_width}" -s | sed "s/^/${list_prefix}/")
readonly FORMATTED_KUBERNETES_VERSIONS
Make breeeze-complete Google Shell Guide compatible (#10708) Also added unit tests for breeze-complete Part of #10576
2020-09-14 11:21:09 +03:00
FORMATTED_KIND_VERSIONS=$(echo "${_breeze_allowed_kind_versions=""}" | tr '\n' ' ' |
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
fold -w "${indented_screen_width}" -s | sed "s/^/${list_prefix}/")
readonly FORMATTED_KIND_VERSIONS
Make breeeze-complete Google Shell Guide compatible (#10708) Also added unit tests for breeze-complete Part of #10576
2020-09-14 11:21:09 +03:00
FORMATTED_HELM_VERSIONS=$(echo "${_breeze_allowed_helm_versions=""}" | tr '\n' ' ' |
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
fold -w "${indented_screen_width}" -s | sed "s/^/${list_prefix}/")
readonly FORMATTED_HELM_VERSIONS
Make breeeze-complete Google Shell Guide compatible (#10708) Also added unit tests for breeze-complete Part of #10576
2020-09-14 11:21:09 +03:00
FORMATTED_KIND_OPERATIONS=$(echo "${_breeze_allowed_kind_operations=""}" | tr '\n' ' ' |
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
fold -w "${indented_screen_width}" -s | sed "s/^/${list_prefix}/")
readonly FORMATTED_KIND_OPERATIONS
Make breeeze-complete Google Shell Guide compatible (#10708) Also added unit tests for breeze-complete Part of #10576
2020-09-14 11:21:09 +03:00
FORMATTED_INSTALL_AIRFLOW_VERSIONS=$(echo "${_breeze_allowed_install_airflow_versions=""}" |
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
tr '\n' ' ' | fold -w "${indented_screen_width}" -s | sed "s/^/${list_prefix}/")
readonly FORMATTED_INSTALL_AIRFLOW_VERSIONS
Make breeeze-complete Google Shell Guide compatible (#10708) Also added unit tests for breeze-complete Part of #10576
2020-09-14 11:21:09 +03:00
FORMATTED_POSTGRES_VERSIONS=$(echo "${_breeze_allowed_postgres_versions=""}" |
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
tr '\n' ' ' | fold -w "${indented_screen_width}" -s | sed "s/^/${list_prefix}/")
readonly FORMATTED_POSTGRES_VERSIONS
Make breeeze-complete Google Shell Guide compatible (#10708) Also added unit tests for breeze-complete Part of #10576
2020-09-14 11:21:09 +03:00
FORMATTED_MYSQL_VERSIONS=$(echo "${_breeze_allowed_mysql_versions=""}" |
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
tr '\n' ' ' | fold -w "${indented_screen_width}" -s | sed "s/^/${list_prefix}/")
readonly FORMATTED_MYSQL_VERSIONS
FORMATTED_DEFAULT_CI_EXTRAS=$(echo "${DEFAULT_CI_EXTRAS=}" |
tr ',' ' ' | fold -w "${indented_screen_width}" -s | sed "s/ /,/g; s/^/${list_prefix}/")
readonly FORMATTED_DEFAULT_CI_EXTRAS
FORMATTED_DEFAULT_PROD_EXTRAS=$(echo "${DEFAULT_PROD_EXTRAS=}" |
tr ',' ' ' | fold -w "${indented_screen_width}" -s | sed "s/ /,/g; s/^/${list_prefix}/")
readonly FORMATTED_DEFAULT_PROD_EXTRAS
[AIRFLOW-6838] Introduce real subcommands for Breeze (#7515) This change introduces sub-commands in breeze tool. It is much needed as we have many commands now and it was difficult to separate commands from flags. Also --help output was very long and unreadable. With this change help it is much easier to discover what breeze can do for you as well as navigate with it. Co-authored-by: Jarek Potiuk <jarek@potiuk.com> Co-authored-by: Kamil Breguła <mik-laj@users.noreply.github.com>
2020-02-25 00:31:50 +03:00
}
[AIRFLOW-6763] Make systems tests ready for backport tests (#7389) We will run system test on back-ported operators for 1.10* series of airflow and for that we need to have support for running system tests using pytest's markers and reading environment variables passed from HOST machine (to pass credentials).  This is the first step to automate system tests execution.
2020-02-21 20:25:32 +03:00
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
#######################################################################################################
#
Add comments to breeze scripts (#8797)
2020-05-10 20:55:40 +03:00
# Prepares usage information for all the commands in Breeze.
# Those usage commands are stored in appropriate environment variables.
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
#
# Created constants:
# USAGE_*
# DETAILED_USAGE_*
#
#######################################################################################################
# shellcheck disable=SC2034,SC2090,SC2089,SC2155
Add packages to function names in bash (#10670) (#10696) Inspired by the Google Shell Guide where they mentioned separating package names with :: I realized that this was one of the missing pieces in the bash scripts of ours. While we already had packages (in libraries folders) it's been difficult to realise which function is where. With introducing packages - equal to the library file name we are *almost* at a level of a structured language - and it's easier to find the functions if you are looking for them. Way easier in fact. Part of #10576 (cherry picked from commit cc551ba793344800d2d396c13d7fd0c8eed97352) (cherry picked from commit 2bba276f0f06a5981bdd7e4f0e7e5ca2fe84f063)
2020-09-02 22:58:37 +03:00
function breeze::prepare_usage() {
[AIRFLOW-6838] Introduce real subcommands for Breeze (#7515) This change introduces sub-commands in breeze tool. It is much needed as we have many commands now and it was difficult to separate commands from flags. Also --help output was very long and unreadable. With this change help it is much easier to discover what breeze can do for you as well as navigate with it. Co-authored-by: Jarek Potiuk <jarek@potiuk.com> Co-authored-by: Kamil Breguła <mik-laj@users.noreply.github.com>
2020-02-25 00:31:50 +03:00
# Note that MacOS uses Bash 3.* and we cannot use associative arrays
export USAGE_SHELL="[Default] Enters interactive shell in the container"
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
readonly USAGE_SHELL
[AIRFLOW-7005] Added exec command to Breeze (#7649)
2020-03-07 22:18:51 +03:00
export USAGE_EXEC="Execs into running breeze container in new terminal"
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
readonly USAGE_EXEC
[AIRFLOW-6838] Introduce real subcommands for Breeze (#7515) This change introduces sub-commands in breeze tool. It is much needed as we have many commands now and it was difficult to separate commands from flags. Also --help output was very long and unreadable. With this change help it is much easier to discover what breeze can do for you as well as navigate with it. Co-authored-by: Jarek Potiuk <jarek@potiuk.com> Co-authored-by: Kamil Breguła <mik-laj@users.noreply.github.com>
2020-02-25 00:31:50 +03:00
export USAGE_BUILD_DOCS="Builds documentation in the container"
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
readonly USAGE_BUILD_DOCS
Add Production Docker image support (#7832)
2020-04-02 20:52:11 +03:00
export USAGE_BUILD_IMAGE="Builds CI or Production docker image"
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
readonly USAGE_BUILD_DOCS
Add Production Docker image support (#7832)
2020-04-02 20:52:11 +03:00
export USAGE_CLEANUP_IMAGE="Cleans up the container image created"
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
readonly USAGE_BUILD_DOCS
[AIRFLOW-6838] Introduce real subcommands for Breeze (#7515) This change introduces sub-commands in breeze tool. It is much needed as we have many commands now and it was difficult to separate commands from flags. Also --help output was very long and unreadable. With this change help it is much easier to discover what breeze can do for you as well as navigate with it. Co-authored-by: Jarek Potiuk <jarek@potiuk.com> Co-authored-by: Kamil Breguła <mik-laj@users.noreply.github.com>
2020-02-25 00:31:50 +03:00
export USAGE_DOCKER_COMPOSE="Executes specified docker-compose command"
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
readonly USAGE_DOCKER_COMPOSE
[AIRFLOW-6838] Introduce real subcommands for Breeze (#7515) This change introduces sub-commands in breeze tool. It is much needed as we have many commands now and it was difficult to separate commands from flags. Also --help output was very long and unreadable. With this change help it is much easier to discover what breeze can do for you as well as navigate with it. Co-authored-by: Jarek Potiuk <jarek@potiuk.com> Co-authored-by: Kamil Breguła <mik-laj@users.noreply.github.com>
2020-02-25 00:31:50 +03:00
export USAGE_FLAGS="Shows all breeze's flags"
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
readonly USAGE_FLAGS
Constraint files are now maintained automatically (#9889) * Constraint files are now maintained automatically * No need to generate requirements when setup.py changes * requirements are kept in separate orphan branches not in main repo * merges to master verify if latest requirements are working and push tested requirements to orphaned branches * we keep history of requirement changes and can label them individually for each version (by constraint-1.10.n tag name) * consistently changed all references to be 'constraints' not 'requirements'
2020-07-20 15:36:03 +03:00
export USAGE_GENERATE_CONSTRAINTS="Generates pinned constraint files"
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
readonly USAGE_GENERATE_CONSTRAINTS
[AIRFLOW-6838] Introduce real subcommands for Breeze (#7515) This change introduces sub-commands in breeze tool. It is much needed as we have many commands now and it was difficult to separate commands from flags. Also --help output was very long and unreadable. With this change help it is much easier to discover what breeze can do for you as well as navigate with it. Co-authored-by: Jarek Potiuk <jarek@potiuk.com> Co-authored-by: Kamil Breguła <mik-laj@users.noreply.github.com>
2020-02-25 00:31:50 +03:00
export USAGE_INITIALIZE_LOCAL_VIRTUALENV="Initializes local virtualenv"
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
readonly USAGE_INITIALIZE_LOCAL_VIRTUALENV
You can push with Breeze as separate command and to cache (#8976) Breeze had --push-images switch to also push images to repo but it was often needed to build and push images separately. We have now a possibility to push an already built image with separate push-image command instead and also you can choose to push to cache registry in GitHub rather than to DockerHub with --registry-cache switch.
2020-06-02 11:24:47 +03:00
export USAGE_PREPARE_BACKPORT_README="Prepares backport packages readme files"
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
readonly USAGE_PREPARE_BACKPORT_README
You can push with Breeze as separate command and to cache (#8976) Breeze had --push-images switch to also push images to repo but it was often needed to build and push images separately. We have now a possibility to push an already built image with separate push-image command instead and also you can choose to push to cache registry in GitHub rather than to DockerHub with --registry-cache switch.
2020-06-02 11:24:47 +03:00
export USAGE_PREPARE_BACKPORT_PACKAGES="Prepares backport packages"
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
readonly USAGE_PREPARE_BACKPORT_PACKAGES
You can push with Breeze as separate command and to cache (#8976) Breeze had --push-images switch to also push images to repo but it was often needed to build and push images separately. We have now a possibility to push an already built image with separate push-image command instead and also you can choose to push to cache registry in GitHub rather than to DockerHub with --registry-cache switch.
2020-06-02 11:24:47 +03:00
export USAGE_PUSH_IMAGE="Pushes images to registry"
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
readonly USAGE_PUSH_IMAGE
Kubernetes Cluster is started on host not in the container (#8265) Tests requiring Kubernetes Cluster are now moved out of the regular CI tests and moved to "kubernetes_tests" folder so that they can be run entirely on host without having the CI image built at all. They use production image to run the tests on KinD cluster and we add tooling to start/stop/deploy the application to the KinD cluster automatically - for both CI testing and local development. This is a pre-requisite to convert the tests to convert the tests to use the official Helm Chart and Docker images or Apache Airflow. It closes #8782
2020-06-03 21:58:38 +03:00
export USAGE_KIND_CLUSTER="Manages KinD cluster on the host"
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
readonly USAGE_KIND_CLUSTER
[AIRFLOW-6838] Introduce real subcommands for Breeze (#7515) This change introduces sub-commands in breeze tool. It is much needed as we have many commands now and it was difficult to separate commands from flags. Also --help output was very long and unreadable. With this change help it is much easier to discover what breeze can do for you as well as navigate with it. Co-authored-by: Jarek Potiuk <jarek@potiuk.com> Co-authored-by: Kamil Breguła <mik-laj@users.noreply.github.com>
2020-02-25 00:31:50 +03:00
export USAGE_SETUP_AUTOCOMPLETE="Sets up autocomplete for breeze"
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
readonly USAGE_SETUP_AUTOCOMPLETE
New Breeze command start-airflow, it replaces the previous flag (#11157)
2020-09-27 19:31:50 +03:00
export USAGE_START_AIRFLOW="Starts Scheduler and Webserver and enters the shell"
readonly USAGE_START_AIRFLOW
Help for breeze commands contain relevant flags. (#8261) * Help for breeze commands contain relevant flags. * fixup! Help for breeze commands contain relevant flags. * fixup! fixup! Help for breeze commands contain relevant flags.
2020-04-13 09:51:33 +03:00
export USAGE_STOP="Stops the docker-compose environment"
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
readonly USAGE_STOP
Help for breeze commands contain relevant flags. (#8261) * Help for breeze commands contain relevant flags. * fixup! Help for breeze commands contain relevant flags. * fixup! fixup! Help for breeze commands contain relevant flags.
2020-04-13 09:51:33 +03:00
export USAGE_RESTART="Stops the docker-compose environment including DB cleanup"
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
readonly USAGE_RESTART
[AIRFLOW-6838] Introduce real subcommands for Breeze (#7515) This change introduces sub-commands in breeze tool. It is much needed as we have many commands now and it was difficult to separate commands from flags. Also --help output was very long and unreadable. With this change help it is much easier to discover what breeze can do for you as well as navigate with it. Co-authored-by: Jarek Potiuk <jarek@potiuk.com> Co-authored-by: Kamil Breguła <mik-laj@users.noreply.github.com>
2020-02-25 00:31:50 +03:00
export USAGE_STATIC_CHECK="Performs selected static check for changed files"
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
readonly USAGE_STATIC_CHECK
[AIRFLOW-6838] Introduce real subcommands for Breeze (#7515) This change introduces sub-commands in breeze tool. It is much needed as we have many commands now and it was difficult to separate commands from flags. Also --help output was very long and unreadable. With this change help it is much easier to discover what breeze can do for you as well as navigate with it. Co-authored-by: Jarek Potiuk <jarek@potiuk.com> Co-authored-by: Kamil Breguła <mik-laj@users.noreply.github.com>
2020-02-25 00:31:50 +03:00
export USAGE_TOGGLE_SUPPRESS_CHEATSHEET="Toggles on/off cheatsheet"
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
readonly USAGE_TOGGLE_SUPPRESS_CHEATSHEET
[AIRFLOW-6838] Introduce real subcommands for Breeze (#7515) This change introduces sub-commands in breeze tool. It is much needed as we have many commands now and it was difficult to separate commands from flags. Also --help output was very long and unreadable. With this change help it is much easier to discover what breeze can do for you as well as navigate with it. Co-authored-by: Jarek Potiuk <jarek@potiuk.com> Co-authored-by: Kamil Breguła <mik-laj@users.noreply.github.com>
2020-02-25 00:31:50 +03:00
export USAGE_TOGGLE_SUPPRESS_ASCIIART="Toggles on/off asciiart"
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
readonly USAGE_TOGGLE_SUPPRESS_ASCIIART
Improve production image iteration speed (#9162) For a long time the way how entrypoint worked in ci scripts was wrong. The way it worked was convoluted and short of black magic. This did not allow to pass multiple test targets and required separate execute command scripts in Breeze. This is all now straightened out and both production and CI image are always using the right entrypoint by default and we can simply pass parameters to the image as usual without escaping strings. This also allowed to remove some breeze commands and change names of several flags in Breeze to make them more meaningful. Both CI and PROD image have now embedded scripts for log cleaning. History of image releases is added for 1.10.10-* alpha quality images.
2020-06-16 13:36:46 +03:00
export USAGE_TESTS="Runs selected tests in the container"
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
readonly USAGE_TESTS
[AIRFLOW-6838] Introduce real subcommands for Breeze (#7515) This change introduces sub-commands in breeze tool. It is much needed as we have many commands now and it was difficult to separate commands from flags. Also --help output was very long and unreadable. With this change help it is much easier to discover what breeze can do for you as well as navigate with it. Co-authored-by: Jarek Potiuk <jarek@potiuk.com> Co-authored-by: Kamil Breguła <mik-laj@users.noreply.github.com>
2020-02-25 00:31:50 +03:00
export USAGE_HELP="Shows this help message"
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
readonly USAGE_HELP
[AIRFLOW-6838] Introduce real subcommands for Breeze (#7515) This change introduces sub-commands in breeze tool. It is much needed as we have many commands now and it was difficult to separate commands from flags. Also --help output was very long and unreadable. With this change help it is much easier to discover what breeze can do for you as well as navigate with it. Co-authored-by: Jarek Potiuk <jarek@potiuk.com> Co-authored-by: Kamil Breguła <mik-laj@users.noreply.github.com>
2020-02-25 00:31:50 +03:00
export USAGE_HELP_ALL="Shows detailed help for all commands and flags"
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
readonly USAGE_HELP_ALL
[AIRFLOW-6838] Introduce real subcommands for Breeze (#7515) This change introduces sub-commands in breeze tool. It is much needed as we have many commands now and it was difficult to separate commands from flags. Also --help output was very long and unreadable. With this change help it is much easier to discover what breeze can do for you as well as navigate with it. Co-authored-by: Jarek Potiuk <jarek@potiuk.com> Co-authored-by: Kamil Breguła <mik-laj@users.noreply.github.com>
2020-02-25 00:31:50 +03:00
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
export DETAILED_USAGE_SHELL="
Improve production image iteration speed (#9162) For a long time the way how entrypoint worked in ci scripts was wrong. The way it worked was convoluted and short of black magic. This did not allow to pass multiple test targets and required separate execute command scripts in Breeze. This is all now straightened out and both production and CI image are always using the right entrypoint by default and we can simply pass parameters to the image as usual without escaping strings. This also allowed to remove some breeze commands and change names of several flags in Breeze to make them more meaningful. Both CI and PROD image have now embedded scripts for log cleaning. History of image releases is added for 1.10.10-* alpha quality images.
2020-06-16 13:36:46 +03:00
${CMDNAME} shell [FLAGS] [-- <EXTRA_ARGS>]
Kubernetes Cluster is started on host not in the container (#8265) Tests requiring Kubernetes Cluster are now moved out of the regular CI tests and moved to "kubernetes_tests" folder so that they can be run entirely on host without having the CI image built at all. They use production image to run the tests on KinD cluster and we add tooling to start/stop/deploy the application to the KinD cluster automatically - for both CI testing and local development. This is a pre-requisite to convert the tests to convert the tests to use the official Helm Chart and Docker images or Apache Airflow. It closes #8782
2020-06-03 21:58:38 +03:00
[AIRFLOW-6838] Introduce real subcommands for Breeze (#7515) This change introduces sub-commands in breeze tool. It is much needed as we have many commands now and it was difficult to separate commands from flags. Also --help output was very long and unreadable. With this change help it is much easier to discover what breeze can do for you as well as navigate with it. Co-authored-by: Jarek Potiuk <jarek@potiuk.com> Co-authored-by: Kamil Breguła <mik-laj@users.noreply.github.com>
2020-02-25 00:31:50 +03:00
This is default subcommand if no subcommand is used.
Help for breeze commands contain relevant flags. (#8261) * Help for breeze commands contain relevant flags. * fixup! Help for breeze commands contain relevant flags. * fixup! fixup! Help for breeze commands contain relevant flags.
2020-04-13 09:51:33 +03:00
Enters interactive shell where you can run all tests, start Airflow webserver, scheduler,
[AIRFLOW-6838] Introduce real subcommands for Breeze (#7515) This change introduces sub-commands in breeze tool. It is much needed as we have many commands now and it was difficult to separate commands from flags. Also --help output was very long and unreadable. With this change help it is much easier to discover what breeze can do for you as well as navigate with it. Co-authored-by: Jarek Potiuk <jarek@potiuk.com> Co-authored-by: Kamil Breguła <mik-laj@users.noreply.github.com>
2020-02-25 00:31:50 +03:00
workers, interact with the database, run DAGs etc. It is the default command if no command
is selected. The shell is executed in the container and in case integrations are chosen,
the integrations will be started as separated docker containers - under the docker-compose
supervision. Local sources are by default mounted to within the container so you can edit
them locally and run tests immediately in the container. Several folders ('files', 'dist')
are also mounted so that you can exchange files between the host and container.
The 'files/airflow-breeze-config/variables.env' file can contain additional variables
and setup. This file is automatically sourced when you enter the container. Database
and webserver ports are forwarded to appropriate database/webserver so that you can
connect to it from your host environment.
Help for breeze commands contain relevant flags. (#8261) * Help for breeze commands contain relevant flags. * fixup! Help for breeze commands contain relevant flags. * fixup! fixup! Help for breeze commands contain relevant flags.
2020-04-13 09:51:33 +03:00
Improve production image iteration speed (#9162) For a long time the way how entrypoint worked in ci scripts was wrong. The way it worked was convoluted and short of black magic. This did not allow to pass multiple test targets and required separate execute command scripts in Breeze. This is all now straightened out and both production and CI image are always using the right entrypoint by default and we can simply pass parameters to the image as usual without escaping strings. This also allowed to remove some breeze commands and change names of several flags in Breeze to make them more meaningful. Both CI and PROD image have now embedded scripts for log cleaning. History of image releases is added for 1.10.10-* alpha quality images.
2020-06-16 13:36:46 +03:00
You can also pass <EXTRA_ARGS> after -- they will be passed as bash parameters, this is
especially useful to pass bash options, for example -c to execute command:
'${CMDNAME} shell -- -c \"ls -la\"'
Update github flags for Breeze (#10384) Part of #10368
2020-08-19 01:53:04 +03:00
'${CMDNAME} -- -c \"ls -la\"'
For DockerHub pull --dockerhub-user and --dockerhub-repo flags can be used to specify
the repository to pull from. For GitHub repository, the --github-repository
flag can be used for the same purpose. You can also use
--github-image-id <COMMIT_SHA>|<RUN_ID> in case you want to pull the image
with specific COMMIT_SHA tag or RUN_ID.
'${CMDNAME} shell \\
--github-image-id 9a621eaa394c0a0a336f8e1b31b35eff4e4ee86e' - pull/use image with SHA
'${CMDNAME} \\
--github-image-id 9a621eaa394c0a0a336f8e1b31b35eff4e4ee86e' - pull/use image with SHA
'${CMDNAME} shell \\
--github-image-id 209845560' - pull/use image with RUN_ID
'${CMDNAME} \\
--github-image-id 209845560' - pull/use image with RUN_ID
Improve production image iteration speed (#9162) For a long time the way how entrypoint worked in ci scripts was wrong. The way it worked was convoluted and short of black magic. This did not allow to pass multiple test targets and required separate execute command scripts in Breeze. This is all now straightened out and both production and CI image are always using the right entrypoint by default and we can simply pass parameters to the image as usual without escaping strings. This also allowed to remove some breeze commands and change names of several flags in Breeze to make them more meaningful. Both CI and PROD image have now embedded scripts for log cleaning. History of image releases is added for 1.10.10-* alpha quality images.
2020-06-16 13:36:46 +03:00
Help for breeze commands contain relevant flags. (#8261) * Help for breeze commands contain relevant flags. * fixup! Help for breeze commands contain relevant flags. * fixup! fixup! Help for breeze commands contain relevant flags.
2020-04-13 09:51:33 +03:00
Flags:
Add packages to function names in bash (#10670) (#10696) Inspired by the Google Shell Guide where they mentioned separating package names with :: I realized that this was one of the missing pieces in the bash scripts of ours. While we already had packages (in libraries folders) it's been difficult to realise which function is where. With introducing packages - equal to the library file name we are *almost* at a level of a structured language - and it's easier to find the functions if you are looking for them. Way easier in fact. Part of #10576 (cherry picked from commit cc551ba793344800d2d396c13d7fd0c8eed97352) (cherry picked from commit 2bba276f0f06a5981bdd7e4f0e7e5ca2fe84f063)
2020-09-02 22:58:37 +03:00
$(breeze::flag_footer)
[AIRFLOW-7005] Added exec command to Breeze (#7649)
2020-03-07 22:18:51 +03:00
"
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
readonly DETAILED_USAGE_SHELL
[AIRFLOW-7005] Added exec command to Breeze (#7649)
2020-03-07 22:18:51 +03:00
export DETAILED_USAGE_EXEC="
Improve production image iteration speed (#9162) For a long time the way how entrypoint worked in ci scripts was wrong. The way it worked was convoluted and short of black magic. This did not allow to pass multiple test targets and required separate execute command scripts in Breeze. This is all now straightened out and both production and CI image are always using the right entrypoint by default and we can simply pass parameters to the image as usual without escaping strings. This also allowed to remove some breeze commands and change names of several flags in Breeze to make them more meaningful. Both CI and PROD image have now embedded scripts for log cleaning. History of image releases is added for 1.10.10-* alpha quality images.
2020-06-16 13:36:46 +03:00
${CMDNAME} exec [-- <EXTRA_ARGS>]
Kubernetes Cluster is started on host not in the container (#8265) Tests requiring Kubernetes Cluster are now moved out of the regular CI tests and moved to "kubernetes_tests" folder so that they can be run entirely on host without having the CI image built at all. They use production image to run the tests on KinD cluster and we add tooling to start/stop/deploy the application to the KinD cluster automatically - for both CI testing and local development. This is a pre-requisite to convert the tests to convert the tests to use the official Helm Chart and Docker images or Apache Airflow. It closes #8782
2020-06-03 21:58:38 +03:00
[AIRFLOW-7005] Added exec command to Breeze (#7649)
2020-03-07 22:18:51 +03:00
Execs into interactive shell to an already running container. The container mus be started
already by breeze shell command. If you are not familiar with tmux, this is the best
way to run multiple processes in the same container at the same time for example scheduler,
webserver, workers, database console and interactive terminal.
[AIRFLOW-6838] Introduce real subcommands for Breeze (#7515) This change introduces sub-commands in breeze tool. It is much needed as we have many commands now and it was difficult to separate commands from flags. Also --help output was very long and unreadable. With this change help it is much easier to discover what breeze can do for you as well as navigate with it. Co-authored-by: Jarek Potiuk <jarek@potiuk.com> Co-authored-by: Kamil Breguła <mik-laj@users.noreply.github.com>
2020-02-25 00:31:50 +03:00
"
export DETAILED_USAGE_BUILD_DOCS="
You can disable spellcheck or documentation when building docs. (#10377) This cleans up the document building process and replaces it with breeze-only. The original instructions with `pip install -e .[doc]` stopped working so there is no point keeping them. Extracted from #10368
2020-08-18 20:40:18 +03:00
${CMDNAME} build-docs [-- <EXTRA_ARGS>]
Kubernetes Cluster is started on host not in the container (#8265) Tests requiring Kubernetes Cluster are now moved out of the regular CI tests and moved to "kubernetes_tests" folder so that they can be run entirely on host without having the CI image built at all. They use production image to run the tests on KinD cluster and we add tooling to start/stop/deploy the application to the KinD cluster automatically - for both CI testing and local development. This is a pre-requisite to convert the tests to convert the tests to use the official Helm Chart and Docker images or Apache Airflow. It closes #8782
2020-06-03 21:58:38 +03:00
Help for breeze commands contain relevant flags. (#8261) * Help for breeze commands contain relevant flags. * fixup! Help for breeze commands contain relevant flags. * fixup! fixup! Help for breeze commands contain relevant flags.
2020-04-13 09:51:33 +03:00
Builds Airflow documentation. The documentation is build inside docker container - to
[AIRFLOW-6838] Introduce real subcommands for Breeze (#7515) This change introduces sub-commands in breeze tool. It is much needed as we have many commands now and it was difficult to separate commands from flags. Also --help output was very long and unreadable. With this change help it is much easier to discover what breeze can do for you as well as navigate with it. Co-authored-by: Jarek Potiuk <jarek@potiuk.com> Co-authored-by: Kamil Breguła <mik-laj@users.noreply.github.com>
2020-02-25 00:31:50 +03:00
maintain the same build environment for everyone. Appropriate sources are mapped from
the host to the container so that latest sources are used. The folders where documentation
You can disable spellcheck or documentation when building docs. (#10377) This cleans up the document building process and replaces it with breeze-only. The original instructions with `pip install -e .[doc]` stopped working so there is no point keeping them. Extracted from #10368
2020-08-18 20:40:18 +03:00
is generated ('docs/_build') are also mounted to the container - this way results of
[AIRFLOW-6838] Introduce real subcommands for Breeze (#7515) This change introduces sub-commands in breeze tool. It is much needed as we have many commands now and it was difficult to separate commands from flags. Also --help output was very long and unreadable. With this change help it is much easier to discover what breeze can do for you as well as navigate with it. Co-authored-by: Jarek Potiuk <jarek@potiuk.com> Co-authored-by: Kamil Breguła <mik-laj@users.noreply.github.com>
2020-02-25 00:31:50 +03:00
the documentation build is available in the host.
You can disable spellcheck or documentation when building docs. (#10377) This cleans up the document building process and replaces it with breeze-only. The original instructions with `pip install -e .[doc]` stopped working so there is no point keeping them. Extracted from #10368
2020-08-18 20:40:18 +03:00
The possible extra args are: --docs-only, --spellcheck-only, --help
[AIRFLOW-6838] Introduce real subcommands for Breeze (#7515) This change introduces sub-commands in breeze tool. It is much needed as we have many commands now and it was difficult to separate commands from flags. Also --help output was very long and unreadable. With this change help it is much easier to discover what breeze can do for you as well as navigate with it. Co-authored-by: Jarek Potiuk <jarek@potiuk.com> Co-authored-by: Kamil Breguła <mik-laj@users.noreply.github.com>
2020-02-25 00:31:50 +03:00
"
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
readonly DETAILED_USAGE_BUILD_DOCS
export DETAILED_USAGE_BUILD_IMAGE="
Kubernetes Cluster is started on host not in the container (#8265) Tests requiring Kubernetes Cluster are now moved out of the regular CI tests and moved to "kubernetes_tests" folder so that they can be run entirely on host without having the CI image built at all. They use production image to run the tests on KinD cluster and we add tooling to start/stop/deploy the application to the KinD cluster automatically - for both CI testing and local development. This is a pre-requisite to convert the tests to convert the tests to use the official Helm Chart and Docker images or Apache Airflow. It closes #8782
2020-06-03 21:58:38 +03:00
${CMDNAME} build-image [FLAGS]
Add Production Docker image support (#7832)
2020-04-02 20:52:11 +03:00
Builds docker image (CI or production) without entering the container. You can pass
additional options to this command, such as '--force-build-image',
More sensible docker caching strategy for Prod images (#9547) Local caching is now default strategy when building the Production image. You can still change it to pulled - similar to CI builds by providing the right build flag and this is what is used in CI by default. The flags in Breeze are now updated to be more eplanatory and friendly (build-cache-*) and a flag for "disabled" cache option is added as well. Also the Dockerfile and Dockerfile.ci files are not needed any more in the docker context. They used to be needed when we built the Kubernetes image in the container, but since we are now using production image directly - we do not need them any nmore. Combining setting the default strategy to local and removing the Dockerfile from the context has the nice effect that you can iterate much faster on the Production image without triggering rebuilds of half of the docker image as soon as the Dockerfile changes.
2020-06-28 18:38:17 +03:00
'--force-pull-image', '--python', '--build-cache-local' or '-build-cache-pulled'
in order to modify build behaviour.
Add Production Docker image support (#7832)
2020-04-02 20:52:11 +03:00
You can also pass '--production-image' flag to build production image rather than CI image.
Help for breeze commands contain relevant flags. (#8261) * Help for breeze commands contain relevant flags. * fixup! Help for breeze commands contain relevant flags. * fixup! fixup! Help for breeze commands contain relevant flags.
2020-04-13 09:51:33 +03:00
Update github flags for Breeze (#10384) Part of #10368
2020-08-19 01:53:04 +03:00
For DockerHub pull --dockerhub-user and --dockerhub-repo flags can be used to specify
the repository to pull from. For GitHub repository, the --github-repository
flag can be used for the same purpose. You can also use
--github-image-id <COMMIT_SHA>|<RUN_ID> in case you want to pull the image with
specific COMMIT_SHA tag or RUN_ID.
Help for breeze commands contain relevant flags. (#8261) * Help for breeze commands contain relevant flags. * fixup! Help for breeze commands contain relevant flags. * fixup! fixup! Help for breeze commands contain relevant flags.
2020-04-13 09:51:33 +03:00
Flags:
Add packages to function names in bash (#10670) (#10696) Inspired by the Google Shell Guide where they mentioned separating package names with :: I realized that this was one of the missing pieces in the bash scripts of ours. While we already had packages (in libraries folders) it's been difficult to realise which function is where. With introducing packages - equal to the library file name we are *almost* at a level of a structured language - and it's easier to find the functions if you are looking for them. Way easier in fact. Part of #10576 (cherry picked from commit cc551ba793344800d2d396c13d7fd0c8eed97352) (cherry picked from commit 2bba276f0f06a5981bdd7e4f0e7e5ca2fe84f063)
2020-09-02 22:58:37 +03:00
$(breeze::flag_airflow_variants)
$(breeze::flag_choose_different_airflow_version)
$(breeze::flag_production_image)
$(breeze::flag_build_docker_images)
$(breeze::flag_pull_push_docker_images)
$(breeze::flag_verbosity)
[AIRFLOW-6838] Introduce real subcommands for Breeze (#7515) This change introduces sub-commands in breeze tool. It is much needed as we have many commands now and it was difficult to separate commands from flags. Also --help output was very long and unreadable. With this change help it is much easier to discover what breeze can do for you as well as navigate with it. Co-authored-by: Jarek Potiuk <jarek@potiuk.com> Co-authored-by: Kamil Breguła <mik-laj@users.noreply.github.com>
2020-02-25 00:31:50 +03:00
"
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
readonly DETAILED_USAGE_BUILD_IMAGE
export DETAILED_USAGE_CLEANUP_IMAGE="
Kubernetes Cluster is started on host not in the container (#8265) Tests requiring Kubernetes Cluster are now moved out of the regular CI tests and moved to "kubernetes_tests" folder so that they can be run entirely on host without having the CI image built at all. They use production image to run the tests on KinD cluster and we add tooling to start/stop/deploy the application to the KinD cluster automatically - for both CI testing and local development. This is a pre-requisite to convert the tests to convert the tests to use the official Helm Chart and Docker images or Apache Airflow. It closes #8782
2020-06-03 21:58:38 +03:00
${CMDNAME} cleanup-image [FLAGS]
[AIRFLOW-6838] Introduce real subcommands for Breeze (#7515) This change introduces sub-commands in breeze tool. It is much needed as we have many commands now and it was difficult to separate commands from flags. Also --help output was very long and unreadable. With this change help it is much easier to discover what breeze can do for you as well as navigate with it. Co-authored-by: Jarek Potiuk <jarek@potiuk.com> Co-authored-by: Kamil Breguła <mik-laj@users.noreply.github.com>
2020-02-25 00:31:50 +03:00
Removes the breeze-related images created in your local docker image cache. This will
not reclaim space in docker cache. You need to 'docker system prune' (optionally
with --all) to reclaim that space.
Help for breeze commands contain relevant flags. (#8261) * Help for breeze commands contain relevant flags. * fixup! Help for breeze commands contain relevant flags. * fixup! fixup! Help for breeze commands contain relevant flags.
2020-04-13 09:51:33 +03:00
Flags:
Add packages to function names in bash (#10670) (#10696) Inspired by the Google Shell Guide where they mentioned separating package names with :: I realized that this was one of the missing pieces in the bash scripts of ours. While we already had packages (in libraries folders) it's been difficult to realise which function is where. With introducing packages - equal to the library file name we are *almost* at a level of a structured language - and it's easier to find the functions if you are looking for them. Way easier in fact. Part of #10576 (cherry picked from commit cc551ba793344800d2d396c13d7fd0c8eed97352) (cherry picked from commit 2bba276f0f06a5981bdd7e4f0e7e5ca2fe84f063)
2020-09-02 22:58:37 +03:00
$(breeze::flag_airflow_variants)
$(breeze::flag_production_image)
$(breeze::flag_verbosity)
[AIRFLOW-6838] Introduce real subcommands for Breeze (#7515) This change introduces sub-commands in breeze tool. It is much needed as we have many commands now and it was difficult to separate commands from flags. Also --help output was very long and unreadable. With this change help it is much easier to discover what breeze can do for you as well as navigate with it. Co-authored-by: Jarek Potiuk <jarek@potiuk.com> Co-authored-by: Kamil Breguła <mik-laj@users.noreply.github.com>
2020-02-25 00:31:50 +03:00
"
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
readonly DETAILED_USAGE_CLEANUP_IMAGE
export DETAILED_USAGE_DOCKER_COMPOSE="
Improve production image iteration speed (#9162) For a long time the way how entrypoint worked in ci scripts was wrong. The way it worked was convoluted and short of black magic. This did not allow to pass multiple test targets and required separate execute command scripts in Breeze. This is all now straightened out and both production and CI image are always using the right entrypoint by default and we can simply pass parameters to the image as usual without escaping strings. This also allowed to remove some breeze commands and change names of several flags in Breeze to make them more meaningful. Both CI and PROD image have now embedded scripts for log cleaning. History of image releases is added for 1.10.10-* alpha quality images.
2020-06-16 13:36:46 +03:00
${CMDNAME} docker-compose [FLAGS] COMMAND [-- <EXTRA_ARGS>]
Kubernetes Cluster is started on host not in the container (#8265) Tests requiring Kubernetes Cluster are now moved out of the regular CI tests and moved to "kubernetes_tests" folder so that they can be run entirely on host without having the CI image built at all. They use production image to run the tests on KinD cluster and we add tooling to start/stop/deploy the application to the KinD cluster automatically - for both CI testing and local development. This is a pre-requisite to convert the tests to convert the tests to use the official Helm Chart and Docker images or Apache Airflow. It closes #8782
2020-06-03 21:58:38 +03:00
[AIRFLOW-6838] Introduce real subcommands for Breeze (#7515) This change introduces sub-commands in breeze tool. It is much needed as we have many commands now and it was difficult to separate commands from flags. Also --help output was very long and unreadable. With this change help it is much easier to discover what breeze can do for you as well as navigate with it. Co-authored-by: Jarek Potiuk <jarek@potiuk.com> Co-authored-by: Kamil Breguła <mik-laj@users.noreply.github.com>
2020-02-25 00:31:50 +03:00
Run docker-compose command instead of entering the environment. Use 'help' as command
to see available commands. The <EXTRA_ARGS> passed after -- are treated
as additional options passed to docker-compose. For example
'${CMDNAME} docker-compose pull -- --ignore-pull-failures'
Help for breeze commands contain relevant flags. (#8261) * Help for breeze commands contain relevant flags. * fixup! Help for breeze commands contain relevant flags. * fixup! fixup! Help for breeze commands contain relevant flags.
2020-04-13 09:51:33 +03:00
Flags:
Add packages to function names in bash (#10670) (#10696) Inspired by the Google Shell Guide where they mentioned separating package names with :: I realized that this was one of the missing pieces in the bash scripts of ours. While we already had packages (in libraries folders) it's been difficult to realise which function is where. With introducing packages - equal to the library file name we are *almost* at a level of a structured language - and it's easier to find the functions if you are looking for them. Way easier in fact. Part of #10576 (cherry picked from commit cc551ba793344800d2d396c13d7fd0c8eed97352) (cherry picked from commit 2bba276f0f06a5981bdd7e4f0e7e5ca2fe84f063)
2020-09-02 22:58:37 +03:00
$(breeze::flag_airflow_variants)
$(breeze::flag_backend_variants)
$(breeze::flag_verbosity)
[AIRFLOW-6838] Introduce real subcommands for Breeze (#7515) This change introduces sub-commands in breeze tool. It is much needed as we have many commands now and it was difficult to separate commands from flags. Also --help output was very long and unreadable. With this change help it is much easier to discover what breeze can do for you as well as navigate with it. Co-authored-by: Jarek Potiuk <jarek@potiuk.com> Co-authored-by: Kamil Breguła <mik-laj@users.noreply.github.com>
2020-02-25 00:31:50 +03:00
"
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
readonly DETAILED_USAGE_DOCKER_COMPOSE
[AIRFLOW-6838] Introduce real subcommands for Breeze (#7515) This change introduces sub-commands in breeze tool. It is much needed as we have many commands now and it was difficult to separate commands from flags. Also --help output was very long and unreadable. With this change help it is much easier to discover what breeze can do for you as well as navigate with it. Co-authored-by: Jarek Potiuk <jarek@potiuk.com> Co-authored-by: Kamil Breguła <mik-laj@users.noreply.github.com>
2020-02-25 00:31:50 +03:00
export DETAILED_USAGE_FLAGS="
Explains in detail all the flags that can be used with breeze.
Requirements now depend on python version (#7841)
2020-03-27 17:27:53 +03:00
"
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
readonly DETAILED_USAGE_FLAGS
export DETAILED_USAGE_PREPARE_BACKPORT_README="
Improve production image iteration speed (#9162) For a long time the way how entrypoint worked in ci scripts was wrong. The way it worked was convoluted and short of black magic. This did not allow to pass multiple test targets and required separate execute command scripts in Breeze. This is all now straightened out and both production and CI image are always using the right entrypoint by default and we can simply pass parameters to the image as usual without escaping strings. This also allowed to remove some breeze commands and change names of several flags in Breeze to make them more meaningful. Both CI and PROD image have now embedded scripts for log cleaning. History of image releases is added for 1.10.10-* alpha quality images.
2020-06-16 13:36:46 +03:00
${CMDNAME} prepare-backport-packages [FLAGS] [YYYY.MM.DD] [PACKAGE_ID ...]
Kubernetes Cluster is started on host not in the container (#8265) Tests requiring Kubernetes Cluster are now moved out of the regular CI tests and moved to "kubernetes_tests" folder so that they can be run entirely on host without having the CI image built at all. They use production image to run the tests on KinD cluster and we add tooling to start/stop/deploy the application to the KinD cluster automatically - for both CI testing and local development. This is a pre-requisite to convert the tests to convert the tests to use the official Helm Chart and Docker images or Apache Airflow. It closes #8782
2020-06-03 21:58:38 +03:00
Added automated release notes generation for backport operators (#8807) We have now mechanism to keep release notes updated for the backport operators in an automated way. It really nicely generates all the necessary information: * summary of requirements for each backport package * list of dependencies (including extras to install them) when package depends on other providers packages * table of new hooks/operators/sensors/protocols/secrets * table of moved hooks/operators/sensors/protocols/secrets with information where they were moved from * changelog of all the changes to the provider package (this will be automatically updated with incremental changelog whenever we decide to release separate packages. The system is fully automated - we will be able to produce release notes automatically (per-package) whenever we decide to release new version of the package in the future.
2020-05-15 20:00:15 +03:00
Prepares README.md files for backport packages. You can provide (after --) optional version
in the form of YYYY.MM.DD, optionally followed by the list of packages to generate readme for.
Fixed name of 20 remaining wrongly named operators. (#8994)
2020-05-26 20:12:21 +03:00
If the first parameter is not formatted as a date, then today is regenerated.
Added automated release notes generation for backport operators (#8807) We have now mechanism to keep release notes updated for the backport operators in an automated way. It really nicely generates all the necessary information: * summary of requirements for each backport package * list of dependencies (including extras to install them) when package depends on other providers packages * table of new hooks/operators/sensors/protocols/secrets * table of moved hooks/operators/sensors/protocols/secrets with information where they were moved from * changelog of all the changes to the provider package (this will be automatically updated with incremental changelog whenever we decide to release separate packages. The system is fully automated - we will be able to produce release notes automatically (per-package) whenever we decide to release new version of the package in the future.
2020-05-15 20:00:15 +03:00
If no packages are specified, readme for all packages are generated.
If no date is specified, current date + 3 days is used (allowing for PMC votes to pass).
Examples:
Release candidate 2 for backport packages 2020.05.20 (#8898) Release candidate 2 for backport packages 2020.05.20
2020-05-19 15:17:22 +03:00
'${CMDNAME} prepare-backport-readme' or
Improve production image iteration speed (#9162) For a long time the way how entrypoint worked in ci scripts was wrong. The way it worked was convoluted and short of black magic. This did not allow to pass multiple test targets and required separate execute command scripts in Breeze. This is all now straightened out and both production and CI image are always using the right entrypoint by default and we can simply pass parameters to the image as usual without escaping strings. This also allowed to remove some breeze commands and change names of several flags in Breeze to make them more meaningful. Both CI and PROD image have now embedded scripts for log cleaning. History of image releases is added for 1.10.10-* alpha quality images.
2020-06-16 13:36:46 +03:00
'${CMDNAME} prepare-backport-readme 2020.05.10' or
'${CMDNAME} prepare-backport-readme 2020.05.10 https google amazon'
Added automated release notes generation for backport operators (#8807) We have now mechanism to keep release notes updated for the backport operators in an automated way. It really nicely generates all the necessary information: * summary of requirements for each backport package * list of dependencies (including extras to install them) when package depends on other providers packages * table of new hooks/operators/sensors/protocols/secrets * table of moved hooks/operators/sensors/protocols/secrets with information where they were moved from * changelog of all the changes to the provider package (this will be automatically updated with incremental changelog whenever we decide to release separate packages. The system is fully automated - we will be able to produce release notes automatically (per-package) whenever we decide to release new version of the package in the future.
2020-05-15 20:00:15 +03:00
General form:
Improve production image iteration speed (#9162) For a long time the way how entrypoint worked in ci scripts was wrong. The way it worked was convoluted and short of black magic. This did not allow to pass multiple test targets and required separate execute command scripts in Breeze. This is all now straightened out and both production and CI image are always using the right entrypoint by default and we can simply pass parameters to the image as usual without escaping strings. This also allowed to remove some breeze commands and change names of several flags in Breeze to make them more meaningful. Both CI and PROD image have now embedded scripts for log cleaning. History of image releases is added for 1.10.10-* alpha quality images.
2020-06-16 13:36:46 +03:00
'${CMDNAME} prepare-backport-readme YYYY.MM.DD <PACKAGE_ID> ...'
Added automated release notes generation for backport operators (#8807) We have now mechanism to keep release notes updated for the backport operators in an automated way. It really nicely generates all the necessary information: * summary of requirements for each backport package * list of dependencies (including extras to install them) when package depends on other providers packages * table of new hooks/operators/sensors/protocols/secrets * table of moved hooks/operators/sensors/protocols/secrets with information where they were moved from * changelog of all the changes to the provider package (this will be automatically updated with incremental changelog whenever we decide to release separate packages. The system is fully automated - we will be able to produce release notes automatically (per-package) whenever we decide to release new version of the package in the future.
2020-05-15 20:00:15 +03:00
* YYYY.MM.DD - is the CALVER version of the package to prepare. Note that this date
cannot be earlier than the already released version (the script will fail if it
will be). It can be set in the future anticipating the future release date.
* <PACKAGE_ID> is usually directory in the airflow/providers folder (for example
'google' but in several cases, it might be one level deeper separated with
'.' for example 'apache.hive'
Flags:
Add packages to function names in bash (#10670) (#10696) Inspired by the Google Shell Guide where they mentioned separating package names with :: I realized that this was one of the missing pieces in the bash scripts of ours. While we already had packages (in libraries folders) it's been difficult to realise which function is where. With introducing packages - equal to the library file name we are *almost* at a level of a structured language - and it's easier to find the functions if you are looking for them. Way easier in fact. Part of #10576 (cherry picked from commit cc551ba793344800d2d396c13d7fd0c8eed97352) (cherry picked from commit 2bba276f0f06a5981bdd7e4f0e7e5ca2fe84f063)
2020-09-02 22:58:37 +03:00
$(breeze::flag_verbosity)
Added automated release notes generation for backport operators (#8807) We have now mechanism to keep release notes updated for the backport operators in an automated way. It really nicely generates all the necessary information: * summary of requirements for each backport package * list of dependencies (including extras to install them) when package depends on other providers packages * table of new hooks/operators/sensors/protocols/secrets * table of moved hooks/operators/sensors/protocols/secrets with information where they were moved from * changelog of all the changes to the provider package (this will be automatically updated with incremental changelog whenever we decide to release separate packages. The system is fully automated - we will be able to produce release notes automatically (per-package) whenever we decide to release new version of the package in the future.
2020-05-15 20:00:15 +03:00
"
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
readonly DETAILED_USAGE_PREPARE_BACKPORT_README
export DETAILED_USAGE_GENERATE_CONSTRAINTS="
Constraint files are now maintained automatically (#9889) * Constraint files are now maintained automatically * No need to generate requirements when setup.py changes * requirements are kept in separate orphan branches not in main repo * merges to master verify if latest requirements are working and push tested requirements to orphaned branches * we keep history of requirement changes and can label them individually for each version (by constraint-1.10.n tag name) * consistently changed all references to be 'constraints' not 'requirements'
2020-07-20 15:36:03 +03:00
${CMDNAME} generate-constraints [FLAGS]
Kubernetes Cluster is started on host not in the container (#8265) Tests requiring Kubernetes Cluster are now moved out of the regular CI tests and moved to "kubernetes_tests" folder so that they can be run entirely on host without having the CI image built at all. They use production image to run the tests on KinD cluster and we add tooling to start/stop/deploy the application to the KinD cluster automatically - for both CI testing and local development. This is a pre-requisite to convert the tests to convert the tests to use the official Helm Chart and Docker images or Apache Airflow. It closes #8782
2020-06-03 21:58:38 +03:00
Constraint files are now maintained automatically (#9889) * Constraint files are now maintained automatically * No need to generate requirements when setup.py changes * requirements are kept in separate orphan branches not in main repo * merges to master verify if latest requirements are working and push tested requirements to orphaned branches * we keep history of requirement changes and can label them individually for each version (by constraint-1.10.n tag name) * consistently changed all references to be 'constraints' not 'requirements'
2020-07-20 15:36:03 +03:00
Generates pinned constraint files from setup.py. Those files are generated in files folder
- separate files for different python version. Those constraint files when pushed to orphan
constraint-master and constraint-1-10 branches are used to generate repeatable
CI builds as well as run repeatable production image builds. You can use those constraints
to predictably install released Airflow versions. This is mainly used to test the constraint
generation - constraints are pushed to the orphan branches by a successful scheduled
CRON job in CI automatically.
Help for breeze commands contain relevant flags. (#8261) * Help for breeze commands contain relevant flags. * fixup! Help for breeze commands contain relevant flags. * fixup! fixup! Help for breeze commands contain relevant flags.
2020-04-13 09:51:33 +03:00
Flags:
Add packages to function names in bash (#10670) (#10696) Inspired by the Google Shell Guide where they mentioned separating package names with :: I realized that this was one of the missing pieces in the bash scripts of ours. While we already had packages (in libraries folders) it's been difficult to realise which function is where. With introducing packages - equal to the library file name we are *almost* at a level of a structured language - and it's easier to find the functions if you are looking for them. Way easier in fact. Part of #10576 (cherry picked from commit cc551ba793344800d2d396c13d7fd0c8eed97352) (cherry picked from commit 2bba276f0f06a5981bdd7e4f0e7e5ca2fe84f063)
2020-09-02 22:58:37 +03:00
$(breeze::flag_airflow_variants)
$(breeze::flag_verbosity)
[AIRFLOW-6838] Introduce real subcommands for Breeze (#7515) This change introduces sub-commands in breeze tool. It is much needed as we have many commands now and it was difficult to separate commands from flags. Also --help output was very long and unreadable. With this change help it is much easier to discover what breeze can do for you as well as navigate with it. Co-authored-by: Jarek Potiuk <jarek@potiuk.com> Co-authored-by: Kamil Breguła <mik-laj@users.noreply.github.com>
2020-02-25 00:31:50 +03:00
"
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
readonly DETAILED_USAGE_GENERATE_CONSTRAINTS
export DETAILED_USAGE_INITIALIZE_LOCAL_VIRTUALENV="
Kubernetes Cluster is started on host not in the container (#8265) Tests requiring Kubernetes Cluster are now moved out of the regular CI tests and moved to "kubernetes_tests" folder so that they can be run entirely on host without having the CI image built at all. They use production image to run the tests on KinD cluster and we add tooling to start/stop/deploy the application to the KinD cluster automatically - for both CI testing and local development. This is a pre-requisite to convert the tests to convert the tests to use the official Helm Chart and Docker images or Apache Airflow. It closes #8782
2020-06-03 21:58:38 +03:00
${CMDNAME} initialize-local-virtualenv [FLAGS]
[AIRFLOW-7067] Pinned version of Apache Airflow (#7730)
2020-03-22 15:34:48 +03:00
Initializes locally created virtualenv installing all dependencies of Airflow
Constraint files are now maintained automatically (#9889) * Constraint files are now maintained automatically * No need to generate requirements when setup.py changes * requirements are kept in separate orphan branches not in main repo * merges to master verify if latest requirements are working and push tested requirements to orphaned branches * we keep history of requirement changes and can label them individually for each version (by constraint-1.10.n tag name) * consistently changed all references to be 'constraints' not 'requirements'
2020-07-20 15:36:03 +03:00
taking into account the constraints for the version specified.
This local virtualenv can be used to aid auto-completion and IDE support as
[AIRFLOW-6838] Introduce real subcommands for Breeze (#7515) This change introduces sub-commands in breeze tool. It is much needed as we have many commands now and it was difficult to separate commands from flags. Also --help output was very long and unreadable. With this change help it is much easier to discover what breeze can do for you as well as navigate with it. Co-authored-by: Jarek Potiuk <jarek@potiuk.com> Co-authored-by: Kamil Breguła <mik-laj@users.noreply.github.com>
2020-02-25 00:31:50 +03:00
well as run unit tests directly from the IDE. You need to have virtualenv
activated before running this command.
Help for breeze commands contain relevant flags. (#8261) * Help for breeze commands contain relevant flags. * fixup! Help for breeze commands contain relevant flags. * fixup! fixup! Help for breeze commands contain relevant flags.
2020-04-13 09:51:33 +03:00
Flags:
Add packages to function names in bash (#10670) (#10696) Inspired by the Google Shell Guide where they mentioned separating package names with :: I realized that this was one of the missing pieces in the bash scripts of ours. While we already had packages (in libraries folders) it's been difficult to realise which function is where. With introducing packages - equal to the library file name we are *almost* at a level of a structured language - and it's easier to find the functions if you are looking for them. Way easier in fact. Part of #10576 (cherry picked from commit cc551ba793344800d2d396c13d7fd0c8eed97352) (cherry picked from commit 2bba276f0f06a5981bdd7e4f0e7e5ca2fe84f063)
2020-09-02 22:58:37 +03:00
$(breeze::flag_airflow_variants)
[AIRFLOW-6838] Introduce real subcommands for Breeze (#7515) This change introduces sub-commands in breeze tool. It is much needed as we have many commands now and it was difficult to separate commands from flags. Also --help output was very long and unreadable. With this change help it is much easier to discover what breeze can do for you as well as navigate with it. Co-authored-by: Jarek Potiuk <jarek@potiuk.com> Co-authored-by: Kamil Breguła <mik-laj@users.noreply.github.com>
2020-02-25 00:31:50 +03:00
"
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
readonly DETAILED_USAGE_INITIALIZE_LOCAL_VIRTUALENV
export DETAILED_USAGE_PREPARE_BACKPORT_PACKAGES="
Improve production image iteration speed (#9162) For a long time the way how entrypoint worked in ci scripts was wrong. The way it worked was convoluted and short of black magic. This did not allow to pass multiple test targets and required separate execute command scripts in Breeze. This is all now straightened out and both production and CI image are always using the right entrypoint by default and we can simply pass parameters to the image as usual without escaping strings. This also allowed to remove some breeze commands and change names of several flags in Breeze to make them more meaningful. Both CI and PROD image have now embedded scripts for log cleaning. History of image releases is added for 1.10.10-* alpha quality images.
2020-06-16 13:36:46 +03:00
${CMDNAME} prepare-backport-packages [FLAGS] [PACKAGE_ID ...]
Kubernetes Cluster is started on host not in the container (#8265) Tests requiring Kubernetes Cluster are now moved out of the regular CI tests and moved to "kubernetes_tests" folder so that they can be run entirely on host without having the CI image built at all. They use production image to run the tests on KinD cluster and we add tooling to start/stop/deploy the application to the KinD cluster automatically - for both CI testing and local development. This is a pre-requisite to convert the tests to convert the tests to use the official Helm Chart and Docker images or Apache Airflow. It closes #8782
2020-06-03 21:58:38 +03:00
Prepares backport packages. You can provide (after --) optional list of packages to prepare.
Added automated release notes generation for backport operators (#8807) We have now mechanism to keep release notes updated for the backport operators in an automated way. It really nicely generates all the necessary information: * summary of requirements for each backport package * list of dependencies (including extras to install them) when package depends on other providers packages * table of new hooks/operators/sensors/protocols/secrets * table of moved hooks/operators/sensors/protocols/secrets with information where they were moved from * changelog of all the changes to the provider package (this will be automatically updated with incremental changelog whenever we decide to release separate packages. The system is fully automated - we will be able to produce release notes automatically (per-package) whenever we decide to release new version of the package in the future.
2020-05-15 20:00:15 +03:00
If no packages are specified, readme for all packages are generated. You can specify optional
Release candidate 2 for backport packages 2020.05.20 (#8898) Release candidate 2 for backport packages 2020.05.20
2020-05-19 15:17:22 +03:00
--version-suffix-for-svn flag to generate rc candidate packages to upload to SVN or
--version-suffix-for-pypi flag to generate rc candidates for PyPI packages.
Added automated release notes generation for backport operators (#8807) We have now mechanism to keep release notes updated for the backport operators in an automated way. It really nicely generates all the necessary information: * summary of requirements for each backport package * list of dependencies (including extras to install them) when package depends on other providers packages * table of new hooks/operators/sensors/protocols/secrets * table of moved hooks/operators/sensors/protocols/secrets with information where they were moved from * changelog of all the changes to the provider package (this will be automatically updated with incremental changelog whenever we decide to release separate packages. The system is fully automated - we will be able to produce release notes automatically (per-package) whenever we decide to release new version of the package in the future.
2020-05-15 20:00:15 +03:00
Examples:
'${CMDNAME} prepare-backport-packages' or
Improve production image iteration speed (#9162) For a long time the way how entrypoint worked in ci scripts was wrong. The way it worked was convoluted and short of black magic. This did not allow to pass multiple test targets and required separate execute command scripts in Breeze. This is all now straightened out and both production and CI image are always using the right entrypoint by default and we can simply pass parameters to the image as usual without escaping strings. This also allowed to remove some breeze commands and change names of several flags in Breeze to make them more meaningful. Both CI and PROD image have now embedded scripts for log cleaning. History of image releases is added for 1.10.10-* alpha quality images.
2020-06-16 13:36:46 +03:00
'${CMDNAME} prepare-backport-packages google' or
'${CMDNAME} prepare-backport-packages --version-suffix-for-svn rc1 http google amazon' or
'${CMDNAME} prepare-backport-packages --version-suffix-for-pypi rc1 http google amazon'
Added automated release notes generation for backport operators (#8807) We have now mechanism to keep release notes updated for the backport operators in an automated way. It really nicely generates all the necessary information: * summary of requirements for each backport package * list of dependencies (including extras to install them) when package depends on other providers packages * table of new hooks/operators/sensors/protocols/secrets * table of moved hooks/operators/sensors/protocols/secrets with information where they were moved from * changelog of all the changes to the provider package (this will be automatically updated with incremental changelog whenever we decide to release separate packages. The system is fully automated - we will be able to produce release notes automatically (per-package) whenever we decide to release new version of the package in the future.
2020-05-15 20:00:15 +03:00
General form:
Release candidate 2 for backport packages 2020.05.20 (#8898) Release candidate 2 for backport packages 2020.05.20
2020-05-19 15:17:22 +03:00
'${CMDNAME} prepare-backport-packages \\
Improve production image iteration speed (#9162) For a long time the way how entrypoint worked in ci scripts was wrong. The way it worked was convoluted and short of black magic. This did not allow to pass multiple test targets and required separate execute command scripts in Breeze. This is all now straightened out and both production and CI image are always using the right entrypoint by default and we can simply pass parameters to the image as usual without escaping strings. This also allowed to remove some breeze commands and change names of several flags in Breeze to make them more meaningful. Both CI and PROD image have now embedded scripts for log cleaning. History of image releases is added for 1.10.10-* alpha quality images.
2020-06-16 13:36:46 +03:00
[--version-suffix-for-svn|--version-suffix-for-pypi] <PACKAGE_ID> ...'
Added automated release notes generation for backport operators (#8807) We have now mechanism to keep release notes updated for the backport operators in an automated way. It really nicely generates all the necessary information: * summary of requirements for each backport package * list of dependencies (including extras to install them) when package depends on other providers packages * table of new hooks/operators/sensors/protocols/secrets * table of moved hooks/operators/sensors/protocols/secrets with information where they were moved from * changelog of all the changes to the provider package (this will be automatically updated with incremental changelog whenever we decide to release separate packages. The system is fully automated - we will be able to produce release notes automatically (per-package) whenever we decide to release new version of the package in the future.
2020-05-15 20:00:15 +03:00
* <PACKAGE_ID> is usually directory in the airflow/providers folder (for example
'google'), but in several cases, it might be one level deeper separated with '.'
for example 'apache.hive'
Flags:
Add packages to function names in bash (#10670) (#10696) Inspired by the Google Shell Guide where they mentioned separating package names with :: I realized that this was one of the missing pieces in the bash scripts of ours. While we already had packages (in libraries folders) it's been difficult to realise which function is where. With introducing packages - equal to the library file name we are *almost* at a level of a structured language - and it's easier to find the functions if you are looking for them. Way easier in fact. Part of #10576 (cherry picked from commit cc551ba793344800d2d396c13d7fd0c8eed97352) (cherry picked from commit 2bba276f0f06a5981bdd7e4f0e7e5ca2fe84f063)
2020-09-02 22:58:37 +03:00
$(breeze::flag_version_suffix)
$(breeze::flag_verbosity)
Added automated release notes generation for backport operators (#8807) We have now mechanism to keep release notes updated for the backport operators in an automated way. It really nicely generates all the necessary information: * summary of requirements for each backport package * list of dependencies (including extras to install them) when package depends on other providers packages * table of new hooks/operators/sensors/protocols/secrets * table of moved hooks/operators/sensors/protocols/secrets with information where they were moved from * changelog of all the changes to the provider package (this will be automatically updated with incremental changelog whenever we decide to release separate packages. The system is fully automated - we will be able to produce release notes automatically (per-package) whenever we decide to release new version of the package in the future.
2020-05-15 20:00:15 +03:00
"
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
readonly DETAILED_USAGE_PREPARE_BACKPORT_PACKAGES
export DETAILED_USAGE_PUSH_IMAGE="
Kubernetes Cluster is started on host not in the container (#8265) Tests requiring Kubernetes Cluster are now moved out of the regular CI tests and moved to "kubernetes_tests" folder so that they can be run entirely on host without having the CI image built at all. They use production image to run the tests on KinD cluster and we add tooling to start/stop/deploy the application to the KinD cluster automatically - for both CI testing and local development. This is a pre-requisite to convert the tests to convert the tests to use the official Helm Chart and Docker images or Apache Airflow. It closes #8782
2020-06-03 21:58:38 +03:00
${CMDNAME} push_image [FLAGS]
You can push with Breeze as separate command and to cache (#8976) Breeze had --push-images switch to also push images to repo but it was often needed to build and push images separately. We have now a possibility to push an already built image with separate push-image command instead and also you can choose to push to cache registry in GitHub rather than to DockerHub with --registry-cache switch.
2020-06-02 11:24:47 +03:00
Pushes images to docker registry. You can push the images to DockerHub registry (default)
Improve production image iteration speed (#9162) For a long time the way how entrypoint worked in ci scripts was wrong. The way it worked was convoluted and short of black magic. This did not allow to pass multiple test targets and required separate execute command scripts in Breeze. This is all now straightened out and both production and CI image are always using the right entrypoint by default and we can simply pass parameters to the image as usual without escaping strings. This also allowed to remove some breeze commands and change names of several flags in Breeze to make them more meaningful. Both CI and PROD image have now embedded scripts for log cleaning. History of image releases is added for 1.10.10-* alpha quality images.
2020-06-16 13:36:46 +03:00
or to the GitHub registry (if --github-registry flag is used).
You can push with Breeze as separate command and to cache (#8976) Breeze had --push-images switch to also push images to repo but it was often needed to build and push images separately. We have now a possibility to push an already built image with separate push-image command instead and also you can choose to push to cache registry in GitHub rather than to DockerHub with --registry-cache switch.
2020-06-02 11:24:47 +03:00
For DockerHub pushes --dockerhub-user and --dockerhub-repo flags can be used to specify
Update github flags for Breeze (#10384) Part of #10368
2020-08-19 01:53:04 +03:00
the repository to push to. For GitHub repository, the --github-repository
flag can be used for the same purpose. You can also add
--github-image-id <COMMIT_SHA>|<RUN_ID> in case you want to push image with specific
SHA tag or run id. In case you specify --github-repository or --github-image-id, you
do not need to specify --github-registry flag.
You can push with Breeze as separate command and to cache (#8976) Breeze had --push-images switch to also push images to repo but it was often needed to build and push images separately. We have now a possibility to push an already built image with separate push-image command instead and also you can choose to push to cache registry in GitHub rather than to DockerHub with --registry-cache switch.
2020-06-02 11:24:47 +03:00
You can also add --production-image flag to switch to production image (default is CI one)
Examples:
'${CMDNAME} push-image' or
'${CMDNAME} push-image --dockerhub-user user' to push to your private registry or
'${CMDNAME} push-image --production-image' - to push production image or
Improve production image iteration speed (#9162) For a long time the way how entrypoint worked in ci scripts was wrong. The way it worked was convoluted and short of black magic. This did not allow to pass multiple test targets and required separate execute command scripts in Breeze. This is all now straightened out and both production and CI image are always using the right entrypoint by default and we can simply pass parameters to the image as usual without escaping strings. This also allowed to remove some breeze commands and change names of several flags in Breeze to make them more meaningful. Both CI and PROD image have now embedded scripts for log cleaning. History of image releases is added for 1.10.10-* alpha quality images.
2020-06-16 13:36:46 +03:00
'${CMDNAME} push-image --github-registry' - to push to GitHub image registry or
Update github flags for Breeze (#10384) Part of #10368
2020-08-19 01:53:04 +03:00
'${CMDNAME} push-image \\
--github-repository user/airflow' - to push to your user's fork
'${CMDNAME} push-image \\
--github-image-id 9a621eaa394c0a0a336f8e1b31b35eff4e4ee86e' - to push with COMMIT_SHA
'${CMDNAME} push-image \\
--github-image-id 209845560' - to push with RUN_ID
You can push with Breeze as separate command and to cache (#8976) Breeze had --push-images switch to also push images to repo but it was often needed to build and push images separately. We have now a possibility to push an already built image with separate push-image command instead and also you can choose to push to cache registry in GitHub rather than to DockerHub with --registry-cache switch.
2020-06-02 11:24:47 +03:00
Flags:
Add packages to function names in bash (#10670) (#10696) Inspired by the Google Shell Guide where they mentioned separating package names with :: I realized that this was one of the missing pieces in the bash scripts of ours. While we already had packages (in libraries folders) it's been difficult to realise which function is where. With introducing packages - equal to the library file name we are *almost* at a level of a structured language - and it's easier to find the functions if you are looking for them. Way easier in fact. Part of #10576 (cherry picked from commit cc551ba793344800d2d396c13d7fd0c8eed97352) (cherry picked from commit 2bba276f0f06a5981bdd7e4f0e7e5ca2fe84f063)
2020-09-02 22:58:37 +03:00
$(breeze::flag_pull_push_docker_images)
$(breeze::flag_verbosity)
You can push with Breeze as separate command and to cache (#8976) Breeze had --push-images switch to also push images to repo but it was often needed to build and push images separately. We have now a possibility to push an already built image with separate push-image command instead and also you can choose to push to cache registry in GitHub rather than to DockerHub with --registry-cache switch.
2020-06-02 11:24:47 +03:00
"
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
readonly DETAILED_USAGE_PUSH_IMAGE
export DETAILED_USAGE_KIND_CLUSTER="
Kubernetes Cluster is started on host not in the container (#8265) Tests requiring Kubernetes Cluster are now moved out of the regular CI tests and moved to "kubernetes_tests" folder so that they can be run entirely on host without having the CI image built at all. They use production image to run the tests on KinD cluster and we add tooling to start/stop/deploy the application to the KinD cluster automatically - for both CI testing and local development. This is a pre-requisite to convert the tests to convert the tests to use the official Helm Chart and Docker images or Apache Airflow. It closes #8782
2020-06-03 21:58:38 +03:00
${CMDNAME} kind-cluster [FLAGS] OPERATION
Manages host-side Kind Kubernetes cluster that is used to run Kubernetes integration tests.
It allows to start/stop/restart/status the Kind Kubernetes cluster and deploy Airflow to it.
Update github flags for Breeze (#10384) Part of #10368
2020-08-19 01:53:04 +03:00
This enables you to run tests inside the breeze environment with latest airflow images.
Kubernetes Cluster is started on host not in the container (#8265) Tests requiring Kubernetes Cluster are now moved out of the regular CI tests and moved to "kubernetes_tests" folder so that they can be run entirely on host without having the CI image built at all. They use production image to run the tests on KinD cluster and we add tooling to start/stop/deploy the application to the KinD cluster automatically - for both CI testing and local development. This is a pre-requisite to convert the tests to convert the tests to use the official Helm Chart and Docker images or Apache Airflow. It closes #8782
2020-06-03 21:58:38 +03:00
Note that in case of deploying airflow, the first step is to rebuild the image and loading it
to the cluster so you can also pass appropriate build image flags that will influence
rebuilding the production image. Operation is one of:
${FORMATTED_KIND_OPERATIONS}
Flags:
Add packages to function names in bash (#10670) (#10696) Inspired by the Google Shell Guide where they mentioned separating package names with :: I realized that this was one of the missing pieces in the bash scripts of ours. While we already had packages (in libraries folders) it's been difficult to realise which function is where. With introducing packages - equal to the library file name we are *almost* at a level of a structured language - and it's easier to find the functions if you are looking for them. Way easier in fact. Part of #10576 (cherry picked from commit cc551ba793344800d2d396c13d7fd0c8eed97352) (cherry picked from commit 2bba276f0f06a5981bdd7e4f0e7e5ca2fe84f063)
2020-09-02 22:58:37 +03:00
$(breeze::flag_airflow_variants)
$(breeze::flag_build_docker_images)
Kubernetes Cluster is started on host not in the container (#8265) Tests requiring Kubernetes Cluster are now moved out of the regular CI tests and moved to "kubernetes_tests" folder so that they can be run entirely on host without having the CI image built at all. They use production image to run the tests on KinD cluster and we add tooling to start/stop/deploy the application to the KinD cluster automatically - for both CI testing and local development. This is a pre-requisite to convert the tests to convert the tests to use the official Helm Chart and Docker images or Apache Airflow. It closes #8782
2020-06-03 21:58:38 +03:00
"
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
readonly DETAILED_USAGE_KIND_CLUSTER
[AIRFLOW-6838] Introduce real subcommands for Breeze (#7515) This change introduces sub-commands in breeze tool. It is much needed as we have many commands now and it was difficult to separate commands from flags. Also --help output was very long and unreadable. With this change help it is much easier to discover what breeze can do for you as well as navigate with it. Co-authored-by: Jarek Potiuk <jarek@potiuk.com> Co-authored-by: Kamil Breguła <mik-laj@users.noreply.github.com>
2020-02-25 00:31:50 +03:00
export DETAILED_USAGE_SETUP_AUTOCOMPLETE="
Kubernetes Cluster is started on host not in the container (#8265) Tests requiring Kubernetes Cluster are now moved out of the regular CI tests and moved to "kubernetes_tests" folder so that they can be run entirely on host without having the CI image built at all. They use production image to run the tests on KinD cluster and we add tooling to start/stop/deploy the application to the KinD cluster automatically - for both CI testing and local development. This is a pre-requisite to convert the tests to convert the tests to use the official Helm Chart and Docker images or Apache Airflow. It closes #8782
2020-06-03 21:58:38 +03:00
${CMDNAME} setup-autocomplete
[AIRFLOW-6838] Introduce real subcommands for Breeze (#7515) This change introduces sub-commands in breeze tool. It is much needed as we have many commands now and it was difficult to separate commands from flags. Also --help output was very long and unreadable. With this change help it is much easier to discover what breeze can do for you as well as navigate with it. Co-authored-by: Jarek Potiuk <jarek@potiuk.com> Co-authored-by: Kamil Breguła <mik-laj@users.noreply.github.com>
2020-02-25 00:31:50 +03:00
Sets up autocomplete for breeze commands. Once you do it you need to re-enter the bash
shell and when typing breeze command <TAB> will provide autocomplete for
parameters and values.
"
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
readonly DETAILED_USAGE_SETUP_AUTOCOMPLETE
New Breeze command start-airflow, it replaces the previous flag (#11157)
2020-09-27 19:31:50 +03:00
export DETAILED_USAGE_START_AIRFLOW="
${CMDNAME} start-airflow
Like the Shell command this will enter the interactive shell, but it will also start
automatically the Scheduler and the Webserver. It will leave you in a tmux session where you
can also observe what is happening in your Airflow.
This is a convenient way to setup a development environment. Your dags will be loaded from the
folder 'files/dags' on your host machine (it could take some times).
If you want to load default connections and example dags you can use the dedicated flags.
Flags:
$(breeze::flag_start_airflow)
"
readonly DETAILED_USAGE_START_AIRFLOW
[AIRFLOW-6972] Shorter frequently used commands in Breeze (#7608)
2020-03-04 03:21:11 +03:00
export DETAILED_USAGE_STOP="
Kubernetes Cluster is started on host not in the container (#8265) Tests requiring Kubernetes Cluster are now moved out of the regular CI tests and moved to "kubernetes_tests" folder so that they can be run entirely on host without having the CI image built at all. They use production image to run the tests on KinD cluster and we add tooling to start/stop/deploy the application to the KinD cluster automatically - for both CI testing and local development. This is a pre-requisite to convert the tests to convert the tests to use the official Helm Chart and Docker images or Apache Airflow. It closes #8782
2020-06-03 21:58:38 +03:00
${CMDNAME} stop
[AIRFLOW-6838] Introduce real subcommands for Breeze (#7515) This change introduces sub-commands in breeze tool. It is much needed as we have many commands now and it was difficult to separate commands from flags. Also --help output was very long and unreadable. With this change help it is much easier to discover what breeze can do for you as well as navigate with it. Co-authored-by: Jarek Potiuk <jarek@potiuk.com> Co-authored-by: Kamil Breguła <mik-laj@users.noreply.github.com>
2020-02-25 00:31:50 +03:00
Brings down running docker compose environment. When you start the environment, the docker
containers will continue running so that startup time is shorter. But they take quite a lot of
memory and CPU. This command stops all running containers from the environment.
[AIRFLOW-6932] Add restart-environment command to Breeze (#7557) When you switch between versions of Aiflow installed, you want to delete the database so that the scripts for resetdb work
2020-02-27 12:44:43 +03:00
"
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
readonly DETAILED_USAGE_STOP
export DETAILED_USAGE_RESTART="
Kubernetes Cluster is started on host not in the container (#8265) Tests requiring Kubernetes Cluster are now moved out of the regular CI tests and moved to "kubernetes_tests" folder so that they can be run entirely on host without having the CI image built at all. They use production image to run the tests on KinD cluster and we add tooling to start/stop/deploy the application to the KinD cluster automatically - for both CI testing and local development. This is a pre-requisite to convert the tests to convert the tests to use the official Helm Chart and Docker images or Apache Airflow. It closes #8782
2020-06-03 21:58:38 +03:00
${CMDNAME} restart [FLAGS]
[AIRFLOW-6932] Add restart-environment command to Breeze (#7557) When you switch between versions of Aiflow installed, you want to delete the database so that the scripts for resetdb work
2020-02-27 12:44:43 +03:00
Restarts running docker compose environment. When you restart the environment, the docker
containers will be restarted. That includes cleaning up the databases. This is
Help for breeze commands contain relevant flags. (#8261) * Help for breeze commands contain relevant flags. * fixup! Help for breeze commands contain relevant flags. * fixup! fixup! Help for breeze commands contain relevant flags.
2020-04-13 09:51:33 +03:00
especially useful if you switch between different versions of Airflow.
Flags:
Add packages to function names in bash (#10670) (#10696) Inspired by the Google Shell Guide where they mentioned separating package names with :: I realized that this was one of the missing pieces in the bash scripts of ours. While we already had packages (in libraries folders) it's been difficult to realise which function is where. With introducing packages - equal to the library file name we are *almost* at a level of a structured language - and it's easier to find the functions if you are looking for them. Way easier in fact. Part of #10576 (cherry picked from commit cc551ba793344800d2d396c13d7fd0c8eed97352) (cherry picked from commit 2bba276f0f06a5981bdd7e4f0e7e5ca2fe84f063)
2020-09-02 22:58:37 +03:00
$(breeze::flag_footer)
[AIRFLOW-6838] Introduce real subcommands for Breeze (#7515) This change introduces sub-commands in breeze tool. It is much needed as we have many commands now and it was difficult to separate commands from flags. Also --help output was very long and unreadable. With this change help it is much easier to discover what breeze can do for you as well as navigate with it. Co-authored-by: Jarek Potiuk <jarek@potiuk.com> Co-authored-by: Kamil Breguła <mik-laj@users.noreply.github.com>
2020-02-25 00:31:50 +03:00
"
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
readonly DETAILED_USAGE_RESTART
[AIRFLOW-6838] Introduce real subcommands for Breeze (#7515) This change introduces sub-commands in breeze tool. It is much needed as we have many commands now and it was difficult to separate commands from flags. Also --help output was very long and unreadable. With this change help it is much easier to discover what breeze can do for you as well as navigate with it. Co-authored-by: Jarek Potiuk <jarek@potiuk.com> Co-authored-by: Kamil Breguła <mik-laj@users.noreply.github.com>
2020-02-25 00:31:50 +03:00
export DETAILED_USAGE_STATIC_CHECK="
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
${CMDNAME} static-check [FLAGS] static_check [-- <EXTRA_ARGS>]
Kubernetes Cluster is started on host not in the container (#8265) Tests requiring Kubernetes Cluster are now moved out of the regular CI tests and moved to "kubernetes_tests" folder so that they can be run entirely on host without having the CI image built at all. They use production image to run the tests on KinD cluster and we add tooling to start/stop/deploy the application to the KinD cluster automatically - for both CI testing and local development. This is a pre-requisite to convert the tests to convert the tests to use the official Helm Chart and Docker images or Apache Airflow. It closes #8782
2020-06-03 21:58:38 +03:00
[AIRFLOW-6838] Introduce real subcommands for Breeze (#7515) This change introduces sub-commands in breeze tool. It is much needed as we have many commands now and it was difficult to separate commands from flags. Also --help output was very long and unreadable. With this change help it is much easier to discover what breeze can do for you as well as navigate with it. Co-authored-by: Jarek Potiuk <jarek@potiuk.com> Co-authored-by: Kamil Breguła <mik-laj@users.noreply.github.com>
2020-02-25 00:31:50 +03:00
Run selected static checks for currently changed files. You should specify static check that
you would like to run or 'all' to run all checks. One of:
[AIRFLOW-6763] Make systems tests ready for backport tests (#7389) We will run system test on back-ported operators for 1.10* series of airflow and for that we need to have support for running system tests using pytest's markers and reading environment variables passed from HOST machine (to pass credentials).  This is the first step to automate system tests execution.
2020-02-21 20:25:32 +03:00
Add Production Docker image support (#7832)
2020-04-02 20:52:11 +03:00
${FORMATTED_STATIC_CHECKS}
[AIRFLOW-3611] Simplified development environment (#4932)
2019-08-27 21:39:36 +03:00
[AIRFLOW-6838] Introduce real subcommands for Breeze (#7515) This change introduces sub-commands in breeze tool. It is much needed as we have many commands now and it was difficult to separate commands from flags. Also --help output was very long and unreadable. With this change help it is much easier to discover what breeze can do for you as well as navigate with it. Co-authored-by: Jarek Potiuk <jarek@potiuk.com> Co-authored-by: Kamil Breguła <mik-laj@users.noreply.github.com>
2020-02-25 00:31:50 +03:00
You can pass extra arguments including options to to the pre-commit framework as
<EXTRA_ARGS> passed after --. For example:
[AIRFLOW-3611] Simplified development environment (#4932)
2019-08-27 21:39:36 +03:00
[AIRFLOW-6838] Introduce real subcommands for Breeze (#7515) This change introduces sub-commands in breeze tool. It is much needed as we have many commands now and it was difficult to separate commands from flags. Also --help output was very long and unreadable. With this change help it is much easier to discover what breeze can do for you as well as navigate with it. Co-authored-by: Jarek Potiuk <jarek@potiuk.com> Co-authored-by: Kamil Breguła <mik-laj@users.noreply.github.com>
2020-02-25 00:31:50 +03:00
'${CMDNAME} static-check mypy' or
'${CMDNAME} static-check mypy -- --files tests/core.py'
Improve production image iteration speed (#9162) For a long time the way how entrypoint worked in ci scripts was wrong. The way it worked was convoluted and short of black magic. This did not allow to pass multiple test targets and required separate execute command scripts in Breeze. This is all now straightened out and both production and CI image are always using the right entrypoint by default and we can simply pass parameters to the image as usual without escaping strings. This also allowed to remove some breeze commands and change names of several flags in Breeze to make them more meaningful. Both CI and PROD image have now embedded scripts for log cleaning. History of image releases is added for 1.10.10-* alpha quality images.
2020-06-16 13:36:46 +03:00
'${CMDNAME} static-check mypy -- --all-files'
[AIRFLOW-3611] Simplified development environment (#4932)
2019-08-27 21:39:36 +03:00
[AIRFLOW-6838] Introduce real subcommands for Breeze (#7515) This change introduces sub-commands in breeze tool. It is much needed as we have many commands now and it was difficult to separate commands from flags. Also --help output was very long and unreadable. With this change help it is much easier to discover what breeze can do for you as well as navigate with it. Co-authored-by: Jarek Potiuk <jarek@potiuk.com> Co-authored-by: Kamil Breguła <mik-laj@users.noreply.github.com>
2020-02-25 00:31:50 +03:00
You can see all the options by adding --help EXTRA_ARG:
[AIRFLOW-3611] Simplified development environment (#4932)
2019-08-27 21:39:36 +03:00
[AIRFLOW-6838] Introduce real subcommands for Breeze (#7515) This change introduces sub-commands in breeze tool. It is much needed as we have many commands now and it was difficult to separate commands from flags. Also --help output was very long and unreadable. With this change help it is much easier to discover what breeze can do for you as well as navigate with it. Co-authored-by: Jarek Potiuk <jarek@potiuk.com> Co-authored-by: Kamil Breguła <mik-laj@users.noreply.github.com>
2020-02-25 00:31:50 +03:00
'${CMDNAME} static-check mypy -- --help'
"
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
readonly DETAILED_USAGE_STATIC_CHECK
export DETAILED_USAGE_TESTS="
Improve production image iteration speed (#9162) For a long time the way how entrypoint worked in ci scripts was wrong. The way it worked was convoluted and short of black magic. This did not allow to pass multiple test targets and required separate execute command scripts in Breeze. This is all now straightened out and both production and CI image are always using the right entrypoint by default and we can simply pass parameters to the image as usual without escaping strings. This also allowed to remove some breeze commands and change names of several flags in Breeze to make them more meaningful. Both CI and PROD image have now embedded scripts for log cleaning. History of image releases is added for 1.10.10-* alpha quality images.
2020-06-16 13:36:46 +03:00
${CMDNAME} tests [FLAGS] [TEST_TARGET ..] [-- <EXTRA_ARGS>]
Kubernetes Cluster is started on host not in the container (#8265) Tests requiring Kubernetes Cluster are now moved out of the regular CI tests and moved to "kubernetes_tests" folder so that they can be run entirely on host without having the CI image built at all. They use production image to run the tests on KinD cluster and we add tooling to start/stop/deploy the application to the KinD cluster automatically - for both CI testing and local development. This is a pre-requisite to convert the tests to convert the tests to use the official Helm Chart and Docker images or Apache Airflow. It closes #8782
2020-06-03 21:58:38 +03:00
[AIRFLOW-6838] Introduce real subcommands for Breeze (#7515) This change introduces sub-commands in breeze tool. It is much needed as we have many commands now and it was difficult to separate commands from flags. Also --help output was very long and unreadable. With this change help it is much easier to discover what breeze can do for you as well as navigate with it. Co-authored-by: Jarek Potiuk <jarek@potiuk.com> Co-authored-by: Kamil Breguła <mik-laj@users.noreply.github.com>
2020-02-25 00:31:50 +03:00
Run the specified unit test target. There might be multiple
targets specified separated with comas. The <EXTRA_ARGS> passed after -- are treated
Improve production image iteration speed (#9162) For a long time the way how entrypoint worked in ci scripts was wrong. The way it worked was convoluted and short of black magic. This did not allow to pass multiple test targets and required separate execute command scripts in Breeze. This is all now straightened out and both production and CI image are always using the right entrypoint by default and we can simply pass parameters to the image as usual without escaping strings. This also allowed to remove some breeze commands and change names of several flags in Breeze to make them more meaningful. Both CI and PROD image have now embedded scripts for log cleaning. History of image releases is added for 1.10.10-* alpha quality images.
2020-06-16 13:36:46 +03:00
as additional options passed to pytest. You can pass 'tests' as target to
run all tests. For example:
[AIRFLOW-6838] Introduce real subcommands for Breeze (#7515) This change introduces sub-commands in breeze tool. It is much needed as we have many commands now and it was difficult to separate commands from flags. Also --help output was very long and unreadable. With this change help it is much easier to discover what breeze can do for you as well as navigate with it. Co-authored-by: Jarek Potiuk <jarek@potiuk.com> Co-authored-by: Kamil Breguła <mik-laj@users.noreply.github.com>
2020-02-25 00:31:50 +03:00
Improve production image iteration speed (#9162) For a long time the way how entrypoint worked in ci scripts was wrong. The way it worked was convoluted and short of black magic. This did not allow to pass multiple test targets and required separate execute command scripts in Breeze. This is all now straightened out and both production and CI image are always using the right entrypoint by default and we can simply pass parameters to the image as usual without escaping strings. This also allowed to remove some breeze commands and change names of several flags in Breeze to make them more meaningful. Both CI and PROD image have now embedded scripts for log cleaning. History of image releases is added for 1.10.10-* alpha quality images.
2020-06-16 13:36:46 +03:00
'${CMDNAME} tests tests/test_core.py -- --logging-level=DEBUG'
'${CMDNAME} tests tests
Help for breeze commands contain relevant flags. (#8261) * Help for breeze commands contain relevant flags. * fixup! Help for breeze commands contain relevant flags. * fixup! fixup! Help for breeze commands contain relevant flags.
2020-04-13 09:51:33 +03:00
Flags:
Add packages to function names in bash (#10670) (#10696) Inspired by the Google Shell Guide where they mentioned separating package names with :: I realized that this was one of the missing pieces in the bash scripts of ours. While we already had packages (in libraries folders) it's been difficult to realise which function is where. With introducing packages - equal to the library file name we are *almost* at a level of a structured language - and it's easier to find the functions if you are looking for them. Way easier in fact. Part of #10576 (cherry picked from commit cc551ba793344800d2d396c13d7fd0c8eed97352) (cherry picked from commit 2bba276f0f06a5981bdd7e4f0e7e5ca2fe84f063)
2020-09-02 22:58:37 +03:00
$(breeze::flag_footer)
[AIRFLOW-6838] Introduce real subcommands for Breeze (#7515) This change introduces sub-commands in breeze tool. It is much needed as we have many commands now and it was difficult to separate commands from flags. Also --help output was very long and unreadable. With this change help it is much easier to discover what breeze can do for you as well as navigate with it. Co-authored-by: Jarek Potiuk <jarek@potiuk.com> Co-authored-by: Kamil Breguła <mik-laj@users.noreply.github.com>
2020-02-25 00:31:50 +03:00
"
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
readonly DETAILED_USAGE_TESTS
[AIRFLOW-6838] Introduce real subcommands for Breeze (#7515) This change introduces sub-commands in breeze tool. It is much needed as we have many commands now and it was difficult to separate commands from flags. Also --help output was very long and unreadable. With this change help it is much easier to discover what breeze can do for you as well as navigate with it. Co-authored-by: Jarek Potiuk <jarek@potiuk.com> Co-authored-by: Kamil Breguła <mik-laj@users.noreply.github.com>
2020-02-25 00:31:50 +03:00
export DETAILED_USAGE_TOGGLE_SUPPRESS_CHEATSHEET="
Kubernetes Cluster is started on host not in the container (#8265) Tests requiring Kubernetes Cluster are now moved out of the regular CI tests and moved to "kubernetes_tests" folder so that they can be run entirely on host without having the CI image built at all. They use production image to run the tests on KinD cluster and we add tooling to start/stop/deploy the application to the KinD cluster automatically - for both CI testing and local development. This is a pre-requisite to convert the tests to convert the tests to use the official Helm Chart and Docker images or Apache Airflow. It closes #8782
2020-06-03 21:58:38 +03:00
${CMDNAME} toggle-suppress-cheatsheet
[AIRFLOW-6838] Introduce real subcommands for Breeze (#7515) This change introduces sub-commands in breeze tool. It is much needed as we have many commands now and it was difficult to separate commands from flags. Also --help output was very long and unreadable. With this change help it is much easier to discover what breeze can do for you as well as navigate with it. Co-authored-by: Jarek Potiuk <jarek@potiuk.com> Co-authored-by: Kamil Breguła <mik-laj@users.noreply.github.com>
2020-02-25 00:31:50 +03:00
Toggles on/off cheatsheet displayed before starting bash shell.
"
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
readonly DETAILED_USAGE_TOGGLE_SUPPRESS_CHEATSHEET
[AIRFLOW-6838] Introduce real subcommands for Breeze (#7515) This change introduces sub-commands in breeze tool. It is much needed as we have many commands now and it was difficult to separate commands from flags. Also --help output was very long and unreadable. With this change help it is much easier to discover what breeze can do for you as well as navigate with it. Co-authored-by: Jarek Potiuk <jarek@potiuk.com> Co-authored-by: Kamil Breguła <mik-laj@users.noreply.github.com>
2020-02-25 00:31:50 +03:00
export DETAILED_USAGE_TOGGLE_SUPPRESS_ASCIIART="
Kubernetes Cluster is started on host not in the container (#8265) Tests requiring Kubernetes Cluster are now moved out of the regular CI tests and moved to "kubernetes_tests" folder so that they can be run entirely on host without having the CI image built at all. They use production image to run the tests on KinD cluster and we add tooling to start/stop/deploy the application to the KinD cluster automatically - for both CI testing and local development. This is a pre-requisite to convert the tests to convert the tests to use the official Helm Chart and Docker images or Apache Airflow. It closes #8782
2020-06-03 21:58:38 +03:00
${CMDNAME} toggle-suppress-asciiart
[AIRFLOW-6838] Introduce real subcommands for Breeze (#7515) This change introduces sub-commands in breeze tool. It is much needed as we have many commands now and it was difficult to separate commands from flags. Also --help output was very long and unreadable. With this change help it is much easier to discover what breeze can do for you as well as navigate with it. Co-authored-by: Jarek Potiuk <jarek@potiuk.com> Co-authored-by: Kamil Breguła <mik-laj@users.noreply.github.com>
2020-02-25 00:31:50 +03:00
Toggles on/off asciiart displayed before starting bash shell.
"
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
readonly DETAILED_USAGE_TOGGLE_SUPPRESS_ASCIIART
[AIRFLOW-6838] Introduce real subcommands for Breeze (#7515) This change introduces sub-commands in breeze tool. It is much needed as we have many commands now and it was difficult to separate commands from flags. Also --help output was very long and unreadable. With this change help it is much easier to discover what breeze can do for you as well as navigate with it. Co-authored-by: Jarek Potiuk <jarek@potiuk.com> Co-authored-by: Kamil Breguła <mik-laj@users.noreply.github.com>
2020-02-25 00:31:50 +03:00
export DETAILED_USAGE_HELP="
Kubernetes Cluster is started on host not in the container (#8265) Tests requiring Kubernetes Cluster are now moved out of the regular CI tests and moved to "kubernetes_tests" folder so that they can be run entirely on host without having the CI image built at all. They use production image to run the tests on KinD cluster and we add tooling to start/stop/deploy the application to the KinD cluster automatically - for both CI testing and local development. This is a pre-requisite to convert the tests to convert the tests to use the official Helm Chart and Docker images or Apache Airflow. It closes #8782
2020-06-03 21:58:38 +03:00
${CMDNAME} help
Shows general help message for all commands.
[AIRFLOW-6838] Introduce real subcommands for Breeze (#7515) This change introduces sub-commands in breeze tool. It is much needed as we have many commands now and it was difficult to separate commands from flags. Also --help output was very long and unreadable. With this change help it is much easier to discover what breeze can do for you as well as navigate with it. Co-authored-by: Jarek Potiuk <jarek@potiuk.com> Co-authored-by: Kamil Breguła <mik-laj@users.noreply.github.com>
2020-02-25 00:31:50 +03:00
"
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
readonly DETAILED_USAGE_HELP
[AIRFLOW-6838] Introduce real subcommands for Breeze (#7515) This change introduces sub-commands in breeze tool. It is much needed as we have many commands now and it was difficult to separate commands from flags. Also --help output was very long and unreadable. With this change help it is much easier to discover what breeze can do for you as well as navigate with it. Co-authored-by: Jarek Potiuk <jarek@potiuk.com> Co-authored-by: Kamil Breguła <mik-laj@users.noreply.github.com>
2020-02-25 00:31:50 +03:00
export DETAILED_USAGE_HELP_ALL="
Kubernetes Cluster is started on host not in the container (#8265) Tests requiring Kubernetes Cluster are now moved out of the regular CI tests and moved to "kubernetes_tests" folder so that they can be run entirely on host without having the CI image built at all. They use production image to run the tests on KinD cluster and we add tooling to start/stop/deploy the application to the KinD cluster automatically - for both CI testing and local development. This is a pre-requisite to convert the tests to convert the tests to use the official Helm Chart and Docker images or Apache Airflow. It closes #8782
2020-06-03 21:58:38 +03:00
${CMDNAME} help-all
[AIRFLOW-6838] Introduce real subcommands for Breeze (#7515) This change introduces sub-commands in breeze tool. It is much needed as we have many commands now and it was difficult to separate commands from flags. Also --help output was very long and unreadable. With this change help it is much easier to discover what breeze can do for you as well as navigate with it. Co-authored-by: Jarek Potiuk <jarek@potiuk.com> Co-authored-by: Kamil Breguła <mik-laj@users.noreply.github.com>
2020-02-25 00:31:50 +03:00
Shows detailed help for all commands and flags.
"
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
readonly DETAILED_USAGE_HELP_ALL
[AIRFLOW-6838] Introduce real subcommands for Breeze (#7515) This change introduces sub-commands in breeze tool. It is much needed as we have many commands now and it was difficult to separate commands from flags. Also --help output was very long and unreadable. With this change help it is much easier to discover what breeze can do for you as well as navigate with it. Co-authored-by: Jarek Potiuk <jarek@potiuk.com> Co-authored-by: Kamil Breguła <mik-laj@users.noreply.github.com>
2020-02-25 00:31:50 +03:00
}
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
# shellcheck enable=all
#######################################################################################################
#
Add comments to breeze scripts (#8797)
2020-05-10 20:55:40 +03:00
# Gets environment variable value converting the lowercase name of command into variable name
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
# Arguments:
# prefix for the variable to add
# name of the variable
# Outputs:
# Writes the capitalized name of the variable to stdout
#######################################################################################################
Add packages to function names in bash (#10670) (#10696) Inspired by the Google Shell Guide where they mentioned separating package names with :: I realized that this was one of the missing pieces in the bash scripts of ours. While we already had packages (in libraries folders) it's been difficult to realise which function is where. With introducing packages - equal to the library file name we are *almost* at a level of a structured language - and it's easier to find the functions if you are looking for them. Way easier in fact. Part of #10576 (cherry picked from commit cc551ba793344800d2d396c13d7fd0c8eed97352) (cherry picked from commit 2bba276f0f06a5981bdd7e4f0e7e5ca2fe84f063)
2020-09-02 22:58:37 +03:00
function breeze::get_variable_from_lowercase_name() {
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
local prefix="${1}"
local name="${2}"
local suffix
suffix="$(echo "${name}" | tr "[:lower:]-" "[:upper:]_")"
local variable_name="${prefix}_${suffix}"
echo "${!variable_name}"
[AIRFLOW-6838] Introduce real subcommands for Breeze (#7515) This change introduces sub-commands in breeze tool. It is much needed as we have many commands now and it was difficult to separate commands from flags. Also --help output was very long and unreadable. With this change help it is much easier to discover what breeze can do for you as well as navigate with it. Co-authored-by: Jarek Potiuk <jarek@potiuk.com> Co-authored-by: Kamil Breguła <mik-laj@users.noreply.github.com>
2020-02-25 00:31:50 +03:00
}
[AIRFLOW-3611] Simplified development environment (#4932)
2019-08-27 21:39:36 +03:00
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
#######################################################################################################
#
Add comments to breeze scripts (#8797)
2020-05-10 20:55:40 +03:00
# Gets usage information from lowercase command
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
# Arguments:
# lowercase command name
# Outputs:
Add packages to function names in bash (#10670) (#10696) Inspired by the Google Shell Guide where they mentioned separating package names with :: I realized that this was one of the missing pieces in the bash scripts of ours. While we already had packages (in libraries folders) it's been difficult to realise which function is where. With introducing packages - equal to the library file name we are *almost* at a level of a structured language - and it's easier to find the functions if you are looking for them. Way easier in fact. Part of #10576 (cherry picked from commit cc551ba793344800d2d396c13d7fd0c8eed97352) (cherry picked from commit 2bba276f0f06a5981bdd7e4f0e7e5ca2fe84f063)
2020-09-02 22:58:37 +03:00
# usage information for the command.
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
#######################################################################################################
Add packages to function names in bash (#10670) (#10696) Inspired by the Google Shell Guide where they mentioned separating package names with :: I realized that this was one of the missing pieces in the bash scripts of ours. While we already had packages (in libraries folders) it's been difficult to realise which function is where. With introducing packages - equal to the library file name we are *almost* at a level of a structured language - and it's easier to find the functions if you are looking for them. Way easier in fact. Part of #10576 (cherry picked from commit cc551ba793344800d2d396c13d7fd0c8eed97352) (cherry picked from commit 2bba276f0f06a5981bdd7e4f0e7e5ca2fe84f063)
2020-09-02 22:58:37 +03:00
function breeze::get_usage() {
breeze::get_variable_from_lowercase_name "USAGE" "${1}"
[AIRFLOW-6838] Introduce real subcommands for Breeze (#7515) This change introduces sub-commands in breeze tool. It is much needed as we have many commands now and it was difficult to separate commands from flags. Also --help output was very long and unreadable. With this change help it is much easier to discover what breeze can do for you as well as navigate with it. Co-authored-by: Jarek Potiuk <jarek@potiuk.com> Co-authored-by: Kamil Breguła <mik-laj@users.noreply.github.com>
2020-02-25 00:31:50 +03:00
}
[AIRFLOW-3611] Simplified development environment (#4932)
2019-08-27 21:39:36 +03:00
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
#######################################################################################################
#
Add comments to breeze scripts (#8797)
2020-05-10 20:55:40 +03:00
# Gets detailed usage information from lowercase command
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
# Arguments:
# lowercase command name
# Outputs:
# Detailed usage information for the command.
#######################################################################################################
Add packages to function names in bash (#10670) (#10696) Inspired by the Google Shell Guide where they mentioned separating package names with :: I realized that this was one of the missing pieces in the bash scripts of ours. While we already had packages (in libraries folders) it's been difficult to realise which function is where. With introducing packages - equal to the library file name we are *almost* at a level of a structured language - and it's easier to find the functions if you are looking for them. Way easier in fact. Part of #10576 (cherry picked from commit cc551ba793344800d2d396c13d7fd0c8eed97352) (cherry picked from commit 2bba276f0f06a5981bdd7e4f0e7e5ca2fe84f063)
2020-09-02 22:58:37 +03:00
function breeze::get_detailed_usage() {
breeze::get_variable_from_lowercase_name "DETAILED_USAGE" "${1}"
[AIRFLOW-6838] Introduce real subcommands for Breeze (#7515) This change introduces sub-commands in breeze tool. It is much needed as we have many commands now and it was difficult to separate commands from flags. Also --help output was very long and unreadable. With this change help it is much easier to discover what breeze can do for you as well as navigate with it. Co-authored-by: Jarek Potiuk <jarek@potiuk.com> Co-authored-by: Kamil Breguła <mik-laj@users.noreply.github.com>
2020-02-25 00:31:50 +03:00
}
[AIRFLOW-3611] Simplified development environment (#4932)
2019-08-27 21:39:36 +03:00
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
#######################################################################################################
#
# Prints general usage information for all commands.
#
# Globals used:
# CMDNAME
Make breeeze-complete Google Shell Guide compatible (#10708) Also added unit tests for breeze-complete Part of #10576
2020-09-14 11:21:09 +03:00
# _breeze_commands
# _breeze_extra_arg_commands
# _breeze_help_commands
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
#
# Outputs:
# General usage information for all commands.
#######################################################################################################
Add packages to function names in bash (#10670) (#10696) Inspired by the Google Shell Guide where they mentioned separating package names with :: I realized that this was one of the missing pieces in the bash scripts of ours. While we already had packages (in libraries folders) it's been difficult to realise which function is where. With introducing packages - equal to the library file name we are *almost* at a level of a structured language - and it's easier to find the functions if you are looking for them. Way easier in fact. Part of #10576 (cherry picked from commit cc551ba793344800d2d396c13d7fd0c8eed97352) (cherry picked from commit 2bba276f0f06a5981bdd7e4f0e7e5ca2fe84f063)
2020-09-02 22:58:37 +03:00
function breeze::usage() {
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
echo "
[AIRFLOW-6838] Introduce real subcommands for Breeze (#7515) This change introduces sub-commands in breeze tool. It is much needed as we have many commands now and it was difficult to separate commands from flags. Also --help output was very long and unreadable. With this change help it is much easier to discover what breeze can do for you as well as navigate with it. Co-authored-by: Jarek Potiuk <jarek@potiuk.com> Co-authored-by: Kamil Breguła <mik-laj@users.noreply.github.com>
2020-02-25 00:31:50 +03:00
Add packages to function names in bash (#10670) (#10696) Inspired by the Google Shell Guide where they mentioned separating package names with :: I realized that this was one of the missing pieces in the bash scripts of ours. While we already had packages (in libraries folders) it's been difficult to realise which function is where. With introducing packages - equal to the library file name we are *almost* at a level of a structured language - and it's easier to find the functions if you are looking for them. Way easier in fact. Part of #10576 (cherry picked from commit cc551ba793344800d2d396c13d7fd0c8eed97352) (cherry picked from commit 2bba276f0f06a5981bdd7e4f0e7e5ca2fe84f063)
2020-09-02 22:58:37 +03:00
usage: ${CMDNAME} [FLAGS] [COMMAND] -- <EXTRA_ARGS>
Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals.
2020-08-31 14:24:53 +03:00
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
By default the script enters the CI container and drops you to bash shell, but you can choose
one of the commands to run specific actions instead.
Add --help after each command to see details:
[AIRFLOW-6838] Introduce real subcommands for Breeze (#7515) This change introduces sub-commands in breeze tool. It is much needed as we have many commands now and it was difficult to separate commands from flags. Also --help output was very long and unreadable. With this change help it is much easier to discover what breeze can do for you as well as navigate with it. Co-authored-by: Jarek Potiuk <jarek@potiuk.com> Co-authored-by: Kamil Breguła <mik-laj@users.noreply.github.com>
2020-02-25 00:31:50 +03:00
Commands without arguments:
"
Make breeeze-complete Google Shell Guide compatible (#10708) Also added unit tests for breeze-complete Part of #10576
2020-09-14 11:21:09 +03:00
for subcommand in ${_breeze_commands}; do
Add packages to function names in bash (#10670) (#10696) Inspired by the Google Shell Guide where they mentioned separating package names with :: I realized that this was one of the missing pieces in the bash scripts of ours. While we already had packages (in libraries folders) it's been difficult to realise which function is where. With introducing packages - equal to the library file name we are *almost* at a level of a structured language - and it's easier to find the functions if you are looking for them. Way easier in fact. Part of #10576 (cherry picked from commit cc551ba793344800d2d396c13d7fd0c8eed97352) (cherry picked from commit 2bba276f0f06a5981bdd7e4f0e7e5ca2fe84f063)
2020-09-02 22:58:37 +03:00
printf " %-40s %s\n" "${subcommand}" "$(breeze::get_usage "${subcommand}")"
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
done
echo "
[AIRFLOW-6838] Introduce real subcommands for Breeze (#7515) This change introduces sub-commands in breeze tool. It is much needed as we have many commands now and it was difficult to separate commands from flags. Also --help output was very long and unreadable. With this change help it is much easier to discover what breeze can do for you as well as navigate with it. Co-authored-by: Jarek Potiuk <jarek@potiuk.com> Co-authored-by: Kamil Breguła <mik-laj@users.noreply.github.com>
2020-02-25 00:31:50 +03:00
Commands with arguments:
"
Make breeeze-complete Google Shell Guide compatible (#10708) Also added unit tests for breeze-complete Part of #10576
2020-09-14 11:21:09 +03:00
# shellcheck disable=SC2154
for subcommand in ${_breeze_extra_arg_commands}; do
Add packages to function names in bash (#10670) (#10696) Inspired by the Google Shell Guide where they mentioned separating package names with :: I realized that this was one of the missing pieces in the bash scripts of ours. While we already had packages (in libraries folders) it's been difficult to realise which function is where. With introducing packages - equal to the library file name we are *almost* at a level of a structured language - and it's easier to find the functions if you are looking for them. Way easier in fact. Part of #10576 (cherry picked from commit cc551ba793344800d2d396c13d7fd0c8eed97352) (cherry picked from commit 2bba276f0f06a5981bdd7e4f0e7e5ca2fe84f063)
2020-09-02 22:58:37 +03:00
printf " %-30s%-10s %s\n" "${subcommand}" "<ARG>" "$(breeze::get_usage "${subcommand}")"
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
done
echo "
[AIRFLOW-6838] Introduce real subcommands for Breeze (#7515) This change introduces sub-commands in breeze tool. It is much needed as we have many commands now and it was difficult to separate commands from flags. Also --help output was very long and unreadable. With this change help it is much easier to discover what breeze can do for you as well as navigate with it. Co-authored-by: Jarek Potiuk <jarek@potiuk.com> Co-authored-by: Kamil Breguła <mik-laj@users.noreply.github.com>
2020-02-25 00:31:50 +03:00
Help commands:
"
Make breeeze-complete Google Shell Guide compatible (#10708) Also added unit tests for breeze-complete Part of #10576
2020-09-14 11:21:09 +03:00
# shellcheck disable=SC2154
for subcommand in ${_breeze_help_commands}; do
Add packages to function names in bash (#10670) (#10696) Inspired by the Google Shell Guide where they mentioned separating package names with :: I realized that this was one of the missing pieces in the bash scripts of ours. While we already had packages (in libraries folders) it's been difficult to realise which function is where. With introducing packages - equal to the library file name we are *almost* at a level of a structured language - and it's easier to find the functions if you are looking for them. Way easier in fact. Part of #10576 (cherry picked from commit cc551ba793344800d2d396c13d7fd0c8eed97352) (cherry picked from commit 2bba276f0f06a5981bdd7e4f0e7e5ca2fe84f063)
2020-09-02 22:58:37 +03:00
printf " %-40s %s\n" "${subcommand}" "$(breeze::get_usage "${subcommand}")"
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
done
echo
[AIRFLOW-6838] Introduce real subcommands for Breeze (#7515) This change introduces sub-commands in breeze tool. It is much needed as we have many commands now and it was difficult to separate commands from flags. Also --help output was very long and unreadable. With this change help it is much easier to discover what breeze can do for you as well as navigate with it. Co-authored-by: Jarek Potiuk <jarek@potiuk.com> Co-authored-by: Kamil Breguła <mik-laj@users.noreply.github.com>
2020-02-25 00:31:50 +03:00
}
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
#######################################################################################################
#
# Prints detailed usage for command specified
#
# Argument:
# Command name.
#
# Outputs:
# Detailed usage information for the command
#######################################################################################################
Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals.
2020-08-31 14:24:53 +03:00
# Prints detailed usage for command specified
Add packages to function names in bash (#10670) (#10696) Inspired by the Google Shell Guide where they mentioned separating package names with :: I realized that this was one of the missing pieces in the bash scripts of ours. While we already had packages (in libraries folders) it's been difficult to realise which function is where. With introducing packages - equal to the library file name we are *almost* at a level of a structured language - and it's easier to find the functions if you are looking for them. Way easier in fact. Part of #10576 (cherry picked from commit cc551ba793344800d2d396c13d7fd0c8eed97352) (cherry picked from commit 2bba276f0f06a5981bdd7e4f0e7e5ca2fe84f063)
2020-09-02 22:58:37 +03:00
function breeze::detailed_usage() {
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
subcommand=${1}
echo "
Help for breeze commands contain relevant flags. (#8261) * Help for breeze commands contain relevant flags. * fixup! Help for breeze commands contain relevant flags. * fixup! fixup! Help for breeze commands contain relevant flags.
2020-04-13 09:51:33 +03:00
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
Detailed usage for command: ${subcommand}
Help for breeze commands contain relevant flags. (#8261) * Help for breeze commands contain relevant flags. * fixup! Help for breeze commands contain relevant flags. * fixup! fixup! Help for breeze commands contain relevant flags.
2020-04-13 09:51:33 +03:00
Add packages to function names in bash (#10670) (#10696) Inspired by the Google Shell Guide where they mentioned separating package names with :: I realized that this was one of the missing pieces in the bash scripts of ours. While we already had packages (in libraries folders) it's been difficult to realise which function is where. With introducing packages - equal to the library file name we are *almost* at a level of a structured language - and it's easier to find the functions if you are looking for them. Way easier in fact. Part of #10576 (cherry picked from commit cc551ba793344800d2d396c13d7fd0c8eed97352) (cherry picked from commit 2bba276f0f06a5981bdd7e4f0e7e5ca2fe84f063)
2020-09-02 22:58:37 +03:00
$(breeze::get_detailed_usage "${subcommand}")
Help for breeze commands contain relevant flags. (#8261) * Help for breeze commands contain relevant flags. * fixup! Help for breeze commands contain relevant flags. * fixup! fixup! Help for breeze commands contain relevant flags.
2020-04-13 09:51:33 +03:00
"
[AIRFLOW-6838] Introduce real subcommands for Breeze (#7515) This change introduces sub-commands in breeze tool. It is much needed as we have many commands now and it was difficult to separate commands from flags. Also --help output was very long and unreadable. With this change help it is much easier to discover what breeze can do for you as well as navigate with it. Co-authored-by: Jarek Potiuk <jarek@potiuk.com> Co-authored-by: Kamil Breguła <mik-laj@users.noreply.github.com>
2020-02-25 00:31:50 +03:00
}
[AIRFLOW-3611] Simplified development environment (#4932)
2019-08-27 21:39:36 +03:00
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
#######################################################################################################
#
# Prints flag footer - common to all commands..
#
# Outputs:
# Footer common for all commands.
#######################################################################################################
Add packages to function names in bash (#10670) (#10696) Inspired by the Google Shell Guide where they mentioned separating package names with :: I realized that this was one of the missing pieces in the bash scripts of ours. While we already had packages (in libraries folders) it's been difficult to realise which function is where. With introducing packages - equal to the library file name we are *almost* at a level of a structured language - and it's easier to find the functions if you are looking for them. Way easier in fact. Part of #10576 (cherry picked from commit cc551ba793344800d2d396c13d7fd0c8eed97352) (cherry picked from commit 2bba276f0f06a5981bdd7e4f0e7e5ca2fe84f063)
2020-09-02 22:58:37 +03:00
function breeze::flag_footer() {
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
echo "
Help for breeze commands contain relevant flags. (#8261) * Help for breeze commands contain relevant flags. * fixup! Help for breeze commands contain relevant flags. * fixup! fixup! Help for breeze commands contain relevant flags.
2020-04-13 09:51:33 +03:00
Run '${CMDNAME} flags' to see all applicable flags.
[AIRFLOW-6838] Introduce real subcommands for Breeze (#7515) This change introduces sub-commands in breeze tool. It is much needed as we have many commands now and it was difficult to separate commands from flags. Also --help output was very long and unreadable. With this change help it is much easier to discover what breeze can do for you as well as navigate with it. Co-authored-by: Jarek Potiuk <jarek@potiuk.com> Co-authored-by: Kamil Breguła <mik-laj@users.noreply.github.com>
2020-02-25 00:31:50 +03:00
"
}
[AIRFLOW-3611] Simplified development environment (#4932)
2019-08-27 21:39:36 +03:00
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
#######################################################################################################
#
Add comments to breeze scripts (#8797)
2020-05-10 20:55:40 +03:00
# Prints flags for different variants of airflow to use
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
#
# Global constants used:
# FORMATTED_PYTHON_MAJOR_MINOR_VERSIONS
#
# Outputs:
# Flag information.
#######################################################################################################
Add packages to function names in bash (#10670) (#10696) Inspired by the Google Shell Guide where they mentioned separating package names with :: I realized that this was one of the missing pieces in the bash scripts of ours. While we already had packages (in libraries folders) it's been difficult to realise which function is where. With introducing packages - equal to the library file name we are *almost* at a level of a structured language - and it's easier to find the functions if you are looking for them. Way easier in fact. Part of #10576 (cherry picked from commit cc551ba793344800d2d396c13d7fd0c8eed97352) (cherry picked from commit 2bba276f0f06a5981bdd7e4f0e7e5ca2fe84f063)
2020-09-02 22:58:37 +03:00
function breeze::flag_airflow_variants() {
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
echo "
Requirements now depend on python version (#7841)
2020-03-27 17:27:53 +03:00
-p, --python <PYTHON_MAJOR_MINOR_VERSION>
[AIRFLOW-3611] Simplified development environment (#4932)
2019-08-27 21:39:36 +03:00
Python version used for the image. This is always major/minor version.
Improve production image iteration speed (#9162) For a long time the way how entrypoint worked in ci scripts was wrong. The way it worked was convoluted and short of black magic. This did not allow to pass multiple test targets and required separate execute command scripts in Breeze. This is all now straightened out and both production and CI image are always using the right entrypoint by default and we can simply pass parameters to the image as usual without escaping strings. This also allowed to remove some breeze commands and change names of several flags in Breeze to make them more meaningful. Both CI and PROD image have now embedded scripts for log cleaning. History of image releases is added for 1.10.10-* alpha quality images.
2020-06-16 13:36:46 +03:00
Note that versions 2.7 and 3.5 are only valid when installing Airflow 1.10 with
--install-airflow-version or --install-airflow-reference flags.
[AIRFLOW-6763] Make systems tests ready for backport tests (#7389) We will run system test on back-ported operators for 1.10* series of airflow and for that we need to have support for running system tests using pytest's markers and reading environment variables passed from HOST machine (to pass credentials).  This is the first step to automate system tests execution.
2020-02-21 20:25:32 +03:00
One of:
Add Production Docker image support (#7832)
2020-04-02 20:52:11 +03:00
${FORMATTED_PYTHON_MAJOR_MINOR_VERSIONS}
Help for breeze commands contain relevant flags. (#8261) * Help for breeze commands contain relevant flags. * fixup! Help for breeze commands contain relevant flags. * fixup! fixup! Help for breeze commands contain relevant flags.
2020-04-13 09:51:33 +03:00
"
}
[AIRFLOW-6763] Make systems tests ready for backport tests (#7389) We will run system test on back-ported operators for 1.10* series of airflow and for that we need to have support for running system tests using pytest's markers and reading environment variables passed from HOST machine (to pass credentials).  This is the first step to automate system tests execution.
2020-02-21 20:25:32 +03:00
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
#######################################################################################################
#
Add comments to breeze scripts (#8797)
2020-05-10 20:55:40 +03:00
# Prints flags for different backend to use
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
#
# Global constants used:
# FORMATTED_BACKENDS
# FORMATTED_POSTGRES_VERSIONS
# FORMATTED_MYSQL_VERSIONS
#
# Outputs:
# Flag information.
#######################################################################################################
Add packages to function names in bash (#10670) (#10696) Inspired by the Google Shell Guide where they mentioned separating package names with :: I realized that this was one of the missing pieces in the bash scripts of ours. While we already had packages (in libraries folders) it's been difficult to realise which function is where. With introducing packages - equal to the library file name we are *almost* at a level of a structured language - and it's easier to find the functions if you are looking for them. Way easier in fact. Part of #10576 (cherry picked from commit cc551ba793344800d2d396c13d7fd0c8eed97352) (cherry picked from commit 2bba276f0f06a5981bdd7e4f0e7e5ca2fe84f063)
2020-09-02 22:58:37 +03:00
function breeze::flag_backend_variants() {
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
echo "
[AIRFLOW-6838] Introduce real subcommands for Breeze (#7515) This change introduces sub-commands in breeze tool. It is much needed as we have many commands now and it was difficult to separate commands from flags. Also --help output was very long and unreadable. With this change help it is much easier to discover what breeze can do for you as well as navigate with it. Co-authored-by: Jarek Potiuk <jarek@potiuk.com> Co-authored-by: Kamil Breguła <mik-laj@users.noreply.github.com>
2020-02-25 00:31:50 +03:00
-b, --backend <BACKEND>
[AIRFLOW-3611] Simplified development environment (#4932)
2019-08-27 21:39:36 +03:00
Backend to use for tests - it determines which database is used.
[AIRFLOW-6763] Make systems tests ready for backport tests (#7389) We will run system test on back-ported operators for 1.10* series of airflow and for that we need to have support for running system tests using pytest's markers and reading environment variables passed from HOST machine (to pass credentials).  This is the first step to automate system tests execution.
2020-02-21 20:25:32 +03:00
One of:
Add Production Docker image support (#7832)
2020-04-02 20:52:11 +03:00
${FORMATTED_BACKENDS}
[AIRFLOW-6763] Make systems tests ready for backport tests (#7389) We will run system test on back-ported operators for 1.10* series of airflow and for that we need to have support for running system tests using pytest's markers and reading environment variables passed from HOST machine (to pass credentials).  This is the first step to automate system tests execution.
2020-02-21 20:25:32 +03:00
Make breeeze-complete Google Shell Guide compatible (#10708) Also added unit tests for breeze-complete Part of #10576
2020-09-14 11:21:09 +03:00
Default: ${_breeze_default_backend:=}
[AIRFLOW-3611] Simplified development environment (#4932)
2019-08-27 21:39:36 +03:00
Help for breeze commands contain relevant flags. (#8261) * Help for breeze commands contain relevant flags. * fixup! Help for breeze commands contain relevant flags. * fixup! fixup! Help for breeze commands contain relevant flags.
2020-04-13 09:51:33 +03:00
--postgres-version <POSTGRES_VERSION>
Postgres version used. One of:
${FORMATTED_POSTGRES_VERSIONS}
--mysql-version <MYSQL_VERSION>
Mysql version used. One of:
${FORMATTED_MYSQL_VERSIONS}
"
}
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
#######################################################################################################
#
Update github flags for Breeze (#10384) Part of #10368
2020-08-19 01:53:04 +03:00
# Prints production image flags
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
#
# Outputs:
# Flag information.
#######################################################################################################
Add packages to function names in bash (#10670) (#10696) Inspired by the Google Shell Guide where they mentioned separating package names with :: I realized that this was one of the missing pieces in the bash scripts of ours. While we already had packages (in libraries folders) it's been difficult to realise which function is where. With introducing packages - equal to the library file name we are *almost* at a level of a structured language - and it's easier to find the functions if you are looking for them. Way easier in fact. Part of #10576 (cherry picked from commit cc551ba793344800d2d396c13d7fd0c8eed97352) (cherry picked from commit 2bba276f0f06a5981bdd7e4f0e7e5ca2fe84f063)
2020-09-02 22:58:37 +03:00
function breeze::flag_production_image() {
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
echo "
Help for breeze commands contain relevant flags. (#8261) * Help for breeze commands contain relevant flags. * fixup! Help for breeze commands contain relevant flags. * fixup! fixup! Help for breeze commands contain relevant flags.
2020-04-13 09:51:33 +03:00
-I, --production-image
Use production image for entering the environment and builds (not for tests).
"
}
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
#######################################################################################################
#
Add comments to breeze scripts (#8797)
2020-05-10 20:55:40 +03:00
# Prints additional breeze action flags
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
#
# Global constants used:
# FORMATTED_INTEGRATIONS
#
#
# Outputs:
# Flag information.
#######################################################################################################
Add packages to function names in bash (#10670) (#10696) Inspired by the Google Shell Guide where they mentioned separating package names with :: I realized that this was one of the missing pieces in the bash scripts of ours. While we already had packages (in libraries folders) it's been difficult to realise which function is where. With introducing packages - equal to the library file name we are *almost* at a level of a structured language - and it's easier to find the functions if you are looking for them. Way easier in fact. Part of #10576 (cherry picked from commit cc551ba793344800d2d396c13d7fd0c8eed97352) (cherry picked from commit 2bba276f0f06a5981bdd7e4f0e7e5ca2fe84f063)
2020-09-02 22:58:37 +03:00
function breeze::flag_breeze_actions() {
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
echo "
[AIRFLOW-7054] Breeze has an option now to reset db at entry (#7710)
2020-03-13 18:08:16 +03:00
-d, --db-reset
Help for breeze commands contain relevant flags. (#8261) * Help for breeze commands contain relevant flags. * fixup! Help for breeze commands contain relevant flags. * fixup! fixup! Help for breeze commands contain relevant flags.
2020-04-13 09:51:33 +03:00
Resets the database at entry to the environment. It will drop all the tables
[AIRFLOW-7054] Breeze has an option now to reset db at entry (#7710)
2020-03-13 18:08:16 +03:00
and data and recreate the DB from scratch even if 'restart' command was not used.
Combined with 'restart' command it enters the environment in the state that is
Help for breeze commands contain relevant flags. (#8261) * Help for breeze commands contain relevant flags. * fixup! Help for breeze commands contain relevant flags. * fixup! fixup! Help for breeze commands contain relevant flags.
2020-04-13 09:51:33 +03:00
ready to start Airflow webserver/scheduler/worker. Without the switch, the database
[AIRFLOW-7054] Breeze has an option now to reset db at entry (#7710)
2020-03-13 18:08:16 +03:00
does not have any tables and you need to run reset db manually.
[AIRFLOW-6838] Introduce real subcommands for Breeze (#7515) This change introduces sub-commands in breeze tool. It is much needed as we have many commands now and it was difficult to separate commands from flags. Also --help output was very long and unreadable. With this change help it is much easier to discover what breeze can do for you as well as navigate with it. Co-authored-by: Jarek Potiuk <jarek@potiuk.com> Co-authored-by: Kamil Breguła <mik-laj@users.noreply.github.com>
2020-02-25 00:31:50 +03:00
-i, --integration <INTEGRATION>
Integration to start during tests - it determines which integrations are started
for integration tests. There can be more than one integration started, or all to
start all integrations. Selected integrations are not saved for future execution.
[AIRFLOW-6763] Make systems tests ready for backport tests (#7389) We will run system test on back-ported operators for 1.10* series of airflow and for that we need to have support for running system tests using pytest's markers and reading environment variables passed from HOST machine (to pass credentials).  This is the first step to automate system tests execution.
2020-02-21 20:25:32 +03:00
One of:
Add Production Docker image support (#7832)
2020-04-02 20:52:11 +03:00
${FORMATTED_INTEGRATIONS}
Flag --start-airflow for breeze (#10837)
2020-09-12 00:26:56 +03:00
Starting breeze will run an init script after the environment is setup (#11029) Added the possibility to run an init script
2020-09-21 13:58:30 +03:00
--init-script <INIT_SCRIPT_FILE>
Initialization script name - Sourced from files/airflow-breeze-config. Default value
New Breeze command start-airflow, it replaces the previous flag (#11157)
2020-09-27 19:31:50 +03:00
init.sh. It will be executed after the environment is configured and started.
Starting breeze will run an init script after the environment is setup (#11029) Added the possibility to run an init script
2020-09-21 13:58:30 +03:00
Help for breeze commands contain relevant flags. (#8261) * Help for breeze commands contain relevant flags. * fixup! Help for breeze commands contain relevant flags. * fixup! fixup! Help for breeze commands contain relevant flags.
2020-04-13 09:51:33 +03:00
"
}
Add Production Docker image support (#7832)
2020-04-02 20:52:11 +03:00
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
#######################################################################################################
#
Add comments to breeze scripts (#8797)
2020-05-10 20:55:40 +03:00
# Prints Kubernetes action flags
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
#
# Global constants used:
# FORMATTED_KUBERNETES_MODE
# FORMATTED_KUBERNETES_VERSIONS
# FORMATTED_KIND_VERSIONS
# FORMATTED_HELM_VERSIONS
#
#
# Outputs:
# Flag information.
#######################################################################################################
Add packages to function names in bash (#10670) (#10696) Inspired by the Google Shell Guide where they mentioned separating package names with :: I realized that this was one of the missing pieces in the bash scripts of ours. While we already had packages (in libraries folders) it's been difficult to realise which function is where. With introducing packages - equal to the library file name we are *almost* at a level of a structured language - and it's easier to find the functions if you are looking for them. Way easier in fact. Part of #10576 (cherry picked from commit cc551ba793344800d2d396c13d7fd0c8eed97352) (cherry picked from commit 2bba276f0f06a5981bdd7e4f0e7e5ca2fe84f063)
2020-09-02 22:58:37 +03:00
function breeze::flag_kubernetes_configuration() {
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
echo "
Kubernetes Cluster is started on host not in the container (#8265) Tests requiring Kubernetes Cluster are now moved out of the regular CI tests and moved to "kubernetes_tests" folder so that they can be run entirely on host without having the CI image built at all. They use production image to run the tests on KinD cluster and we add tooling to start/stop/deploy the application to the KinD cluster automatically - for both CI testing and local development. This is a pre-requisite to convert the tests to convert the tests to use the official Helm Chart and Docker images or Apache Airflow. It closes #8782
2020-06-03 21:58:38 +03:00
Configuration for the KinD Kubernetes cluster and tests:
[AIRFLOW-6838] Introduce real subcommands for Breeze (#7515) This change introduces sub-commands in breeze tool. It is much needed as we have many commands now and it was difficult to separate commands from flags. Also --help output was very long and unreadable. With this change help it is much easier to discover what breeze can do for you as well as navigate with it. Co-authored-by: Jarek Potiuk <jarek@potiuk.com> Co-authored-by: Kamil Breguła <mik-laj@users.noreply.github.com>
2020-02-25 00:31:50 +03:00
-K, --kubernetes-mode <KUBERNETES_MODE>
Mount gcloud kubeconifg to breeze (#10439) * Mount gcloud kubeconifg to breeze * Update scripts/ci/docker-compose/forward-credentials.yml Co-authored-by: Tomek Urbaszek <tomasz.urbaszek@polidea.com> Co-authored-by: Jarek Potiuk <jarek@potiuk.com>
2020-08-21 22:22:32 +03:00
Kubernetes mode - only used in case one of kind-cluster commands is used.
[AIRFLOW-6763] Make systems tests ready for backport tests (#7389) We will run system test on back-ported operators for 1.10* series of airflow and for that we need to have support for running system tests using pytest's markers and reading environment variables passed from HOST machine (to pass credentials).  This is the first step to automate system tests execution.
2020-02-21 20:25:32 +03:00
One of:
Add Production Docker image support (#7832)
2020-04-02 20:52:11 +03:00
${FORMATTED_KUBERNETES_MODES}
[AIRFLOW-6763] Make systems tests ready for backport tests (#7389) We will run system test on back-ported operators for 1.10* series of airflow and for that we need to have support for running system tests using pytest's markers and reading environment variables passed from HOST machine (to pass credentials).  This is the first step to automate system tests execution.
2020-02-21 20:25:32 +03:00
Make breeeze-complete Google Shell Guide compatible (#10708) Also added unit tests for breeze-complete Part of #10576
2020-09-14 11:21:09 +03:00
Default: ${_breeze_default_kubernetes_mode:=}
[AIRFLOW-3611] Simplified development environment (#4932)
2019-08-27 21:39:36 +03:00
[AIRFLOW-5704] Improve Kind Kubernetes scripts for local testing (#6516) * Fixed problem that Kubernetes tests were testing latest master rather than what came from the local sources. * Kind (Kubernetes in Dcocker) is run in the same Docker as Breeze env * Moved Kubernetes scripts to 'in_container' dir where they belong now * Kubernetes cluster is reused until it is stopped * Kubernetes image is build from image in docker already + mounted sources * Kubectl version name is corrected in the Dockerfile * KUBERNETES_VERSION can now be used to select Kubernetes version * Running kubernetes scripts is now easy in Breeze * We can start/recreate/stop cluster using --<ACTION>-kind-cluster * Instructions on how to run Kubernetes tests are updated * The old "bare" environment is replaced by --no-deps switch
2020-01-11 18:25:19 +03:00
-V, --kubernetes-version <KUBERNETES_VERSION>
Mount gcloud kubeconifg to breeze (#10439) * Mount gcloud kubeconifg to breeze * Update scripts/ci/docker-compose/forward-credentials.yml Co-authored-by: Tomek Urbaszek <tomasz.urbaszek@polidea.com> Co-authored-by: Jarek Potiuk <jarek@potiuk.com>
2020-08-21 22:22:32 +03:00
Kubernetes version - only used in case one of kind-cluster commands is used.
[AIRFLOW-6763] Make systems tests ready for backport tests (#7389) We will run system test on back-ported operators for 1.10* series of airflow and for that we need to have support for running system tests using pytest's markers and reading environment variables passed from HOST machine (to pass credentials).  This is the first step to automate system tests execution.
2020-02-21 20:25:32 +03:00
One of:
Add Production Docker image support (#7832)
2020-04-02 20:52:11 +03:00
${FORMATTED_KUBERNETES_VERSIONS}
[AIRFLOW-6763] Make systems tests ready for backport tests (#7389) We will run system test on back-ported operators for 1.10* series of airflow and for that we need to have support for running system tests using pytest's markers and reading environment variables passed from HOST machine (to pass credentials).  This is the first step to automate system tests execution.
2020-02-21 20:25:32 +03:00
Make breeeze-complete Google Shell Guide compatible (#10708) Also added unit tests for breeze-complete Part of #10576
2020-09-14 11:21:09 +03:00
Default: ${_breeze_default_kubernetes_version:=}
Switches to Helm Chart for Kubernetes tests (#9468) The Kubernetes tests are now run using Helm chart rather than the custom templates we used to have. The Helm Chart uses locally build production image so the tests are testing not only Airflow but also Helm Chart and a Production image - all at the same time. Later on we will add more tests covering more functionalities of both Helm Chart and Production Image. This is the first step to get all of those bundle together and become testable. This change introduces also 'shell' sub-command for Breeze's kind-cluster command and EMBEDDED_DAGS build args for production image - both of them useful to run the Kubernetes tests more easily - without building two images and with an easy-to-iterate-over-tests shell command - which works without any other development environment. Co-authored-by: Jarek Potiuk <jarek@potiuk.com> Co-authored-by: Daniel Imberman <daniel@astronomer.io>
2020-07-01 15:50:30 +03:00
--kind-version <KIND_VERSION>
Mount gcloud kubeconifg to breeze (#10439) * Mount gcloud kubeconifg to breeze * Update scripts/ci/docker-compose/forward-credentials.yml Co-authored-by: Tomek Urbaszek <tomasz.urbaszek@polidea.com> Co-authored-by: Jarek Potiuk <jarek@potiuk.com>
2020-08-21 22:22:32 +03:00
Kind version - only used in case one of kind-cluster commands is used.
Switches to Helm Chart for Kubernetes tests (#9468) The Kubernetes tests are now run using Helm chart rather than the custom templates we used to have. The Helm Chart uses locally build production image so the tests are testing not only Airflow but also Helm Chart and a Production image - all at the same time. Later on we will add more tests covering more functionalities of both Helm Chart and Production Image. This is the first step to get all of those bundle together and become testable. This change introduces also 'shell' sub-command for Breeze's kind-cluster command and EMBEDDED_DAGS build args for production image - both of them useful to run the Kubernetes tests more easily - without building two images and with an easy-to-iterate-over-tests shell command - which works without any other development environment. Co-authored-by: Jarek Potiuk <jarek@potiuk.com> Co-authored-by: Daniel Imberman <daniel@astronomer.io>
2020-07-01 15:50:30 +03:00
One of:
${FORMATTED_KIND_VERSIONS}
Make breeeze-complete Google Shell Guide compatible (#10708) Also added unit tests for breeze-complete Part of #10576
2020-09-14 11:21:09 +03:00
Default: ${_breeze_default_kind_version:=}
Switches to Helm Chart for Kubernetes tests (#9468) The Kubernetes tests are now run using Helm chart rather than the custom templates we used to have. The Helm Chart uses locally build production image so the tests are testing not only Airflow but also Helm Chart and a Production image - all at the same time. Later on we will add more tests covering more functionalities of both Helm Chart and Production Image. This is the first step to get all of those bundle together and become testable. This change introduces also 'shell' sub-command for Breeze's kind-cluster command and EMBEDDED_DAGS build args for production image - both of them useful to run the Kubernetes tests more easily - without building two images and with an easy-to-iterate-over-tests shell command - which works without any other development environment. Co-authored-by: Jarek Potiuk <jarek@potiuk.com> Co-authored-by: Daniel Imberman <daniel@astronomer.io>
2020-07-01 15:50:30 +03:00
--helm-version <HELM_VERSION>
Mount gcloud kubeconifg to breeze (#10439) * Mount gcloud kubeconifg to breeze * Update scripts/ci/docker-compose/forward-credentials.yml Co-authored-by: Tomek Urbaszek <tomasz.urbaszek@polidea.com> Co-authored-by: Jarek Potiuk <jarek@potiuk.com>
2020-08-21 22:22:32 +03:00
Helm version - only used in case one of kind-cluster commands is used.
Switches to Helm Chart for Kubernetes tests (#9468) The Kubernetes tests are now run using Helm chart rather than the custom templates we used to have. The Helm Chart uses locally build production image so the tests are testing not only Airflow but also Helm Chart and a Production image - all at the same time. Later on we will add more tests covering more functionalities of both Helm Chart and Production Image. This is the first step to get all of those bundle together and become testable. This change introduces also 'shell' sub-command for Breeze's kind-cluster command and EMBEDDED_DAGS build args for production image - both of them useful to run the Kubernetes tests more easily - without building two images and with an easy-to-iterate-over-tests shell command - which works without any other development environment. Co-authored-by: Jarek Potiuk <jarek@potiuk.com> Co-authored-by: Daniel Imberman <daniel@astronomer.io>
2020-07-01 15:50:30 +03:00
One of:
${FORMATTED_HELM_VERSIONS}
Make breeeze-complete Google Shell Guide compatible (#10708) Also added unit tests for breeze-complete Part of #10576
2020-09-14 11:21:09 +03:00
Default: ${_breeze_default_helm_version:=}
Switches to Helm Chart for Kubernetes tests (#9468) The Kubernetes tests are now run using Helm chart rather than the custom templates we used to have. The Helm Chart uses locally build production image so the tests are testing not only Airflow but also Helm Chart and a Production image - all at the same time. Later on we will add more tests covering more functionalities of both Helm Chart and Production Image. This is the first step to get all of those bundle together and become testable. This change introduces also 'shell' sub-command for Breeze's kind-cluster command and EMBEDDED_DAGS build args for production image - both of them useful to run the Kubernetes tests more easily - without building two images and with an easy-to-iterate-over-tests shell command - which works without any other development environment. Co-authored-by: Jarek Potiuk <jarek@potiuk.com> Co-authored-by: Daniel Imberman <daniel@astronomer.io>
2020-07-01 15:50:30 +03:00
Help for breeze commands contain relevant flags. (#8261) * Help for breeze commands contain relevant flags. * fixup! Help for breeze commands contain relevant flags. * fixup! fixup! Help for breeze commands contain relevant flags.
2020-04-13 09:51:33 +03:00
"
}
[AIRFLOW-5704] Improve Kind Kubernetes scripts for local testing (#6516) * Fixed problem that Kubernetes tests were testing latest master rather than what came from the local sources. * Kind (Kubernetes in Dcocker) is run in the same Docker as Breeze env * Moved Kubernetes scripts to 'in_container' dir where they belong now * Kubernetes cluster is reused until it is stopped * Kubernetes image is build from image in docker already + mounted sources * Kubectl version name is corrected in the Dockerfile * KUBERNETES_VERSION can now be used to select Kubernetes version * Running kubernetes scripts is now easy in Breeze * We can start/recreate/stop cluster using --<ACTION>-kind-cluster * Instructions on how to run Kubernetes tests are updated * The old "bare" environment is replaced by --no-deps switch
2020-01-11 18:25:19 +03:00
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
#######################################################################################################
#
Add comments to breeze scripts (#8797)
2020-05-10 20:55:40 +03:00
# Prints flags that determine what is the source mounting scheme
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
#
# Outputs:
# Flag information.
#######################################################################################################
Add packages to function names in bash (#10670) (#10696) Inspired by the Google Shell Guide where they mentioned separating package names with :: I realized that this was one of the missing pieces in the bash scripts of ours. While we already had packages (in libraries folders) it's been difficult to realise which function is where. With introducing packages - equal to the library file name we are *almost* at a level of a structured language - and it's easier to find the functions if you are looking for them. Way easier in fact. Part of #10576 (cherry picked from commit cc551ba793344800d2d396c13d7fd0c8eed97352) (cherry picked from commit 2bba276f0f06a5981bdd7e4f0e7e5ca2fe84f063)
2020-09-02 22:58:37 +03:00
function breeze::flag_local_file_mounting() {
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
echo "
[AIRFLOW-5828] Move build logic out from hooks/build (#7618) This is the final step of simplifying the Breeze scripts by moving all the logic out from Travis' hooks/build
2020-03-07 18:09:06 +03:00
-l, --skip-mounting-local-sources
[AIRFLOW-3611] Simplified development environment (#4932)
2019-08-27 21:39:36 +03:00
Skips mounting local volume with sources - you get exactly what is in the
Help for breeze commands contain relevant flags. (#8261) * Help for breeze commands contain relevant flags. * fixup! Help for breeze commands contain relevant flags. * fixup! fixup! Help for breeze commands contain relevant flags.
2020-04-13 09:51:33 +03:00
docker image rather than your current local sources of Airflow.
"
}
[AIRFLOW-6763] Make systems tests ready for backport tests (#7389) We will run system test on back-ported operators for 1.10* series of airflow and for that we need to have support for running system tests using pytest's markers and reading environment variables passed from HOST machine (to pass credentials).  This is the first step to automate system tests execution.
2020-02-21 20:25:32 +03:00
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
#######################################################################################################
#
Add comments to breeze scripts (#8797)
2020-05-10 20:55:40 +03:00
# Prints flags that allow to choose different airflow variants
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
#
# Global constants used:
# FORMATTED_INSTALL_AIRFLOW_VERSIONS
#
#
# Outputs:
# Flag information.
#######################################################################################################
Add packages to function names in bash (#10670) (#10696) Inspired by the Google Shell Guide where they mentioned separating package names with :: I realized that this was one of the missing pieces in the bash scripts of ours. While we already had packages (in libraries folders) it's been difficult to realise which function is where. With introducing packages - equal to the library file name we are *almost* at a level of a structured language - and it's easier to find the functions if you are looking for them. Way easier in fact. Part of #10576 (cherry picked from commit cc551ba793344800d2d396c13d7fd0c8eed97352) (cherry picked from commit 2bba276f0f06a5981bdd7e4f0e7e5ca2fe84f063)
2020-09-02 22:58:37 +03:00
function breeze::flag_choose_different_airflow_version() {
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
echo "
[AIRFLOW-6838] Introduce real subcommands for Breeze (#7515) This change introduces sub-commands in breeze tool. It is much needed as we have many commands now and it was difficult to separate commands from flags. Also --help output was very long and unreadable. With this change help it is much easier to discover what breeze can do for you as well as navigate with it. Co-authored-by: Jarek Potiuk <jarek@potiuk.com> Co-authored-by: Kamil Breguła <mik-laj@users.noreply.github.com>
2020-02-25 00:31:50 +03:00
-a, --install-airflow-version <INSTALL_AIRFLOW_VERSION>
Help for breeze commands contain relevant flags. (#8261) * Help for breeze commands contain relevant flags. * fixup! Help for breeze commands contain relevant flags. * fixup! fixup! Help for breeze commands contain relevant flags.
2020-04-13 09:51:33 +03:00
If specified, installs Airflow directly from PIP released version. This happens at
image building time in production image and at container entering time for CI image. One of:
Add Production Docker image support (#7832)
2020-04-02 20:52:11 +03:00
${FORMATTED_INSTALL_AIRFLOW_VERSIONS}
-t, --install-airflow-reference <INSTALL_AIRFLOW_REFERENCE>
Help for breeze commands contain relevant flags. (#8261) * Help for breeze commands contain relevant flags. * fixup! Help for breeze commands contain relevant flags. * fixup! fixup! Help for breeze commands contain relevant flags.
2020-04-13 09:51:33 +03:00
If specified, installs Airflow directly from reference in GitHub. This happens at
image building time in production image and at container entering time for CI image.
"
}
[AIRFLOW-3611] Simplified development environment (#4932)
2019-08-27 21:39:36 +03:00
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
#######################################################################################################
#
Add comments to breeze scripts (#8797)
2020-05-10 20:55:40 +03:00
# Prints flags that allow to set assumed answers to questions
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
#
# Outputs:
# Flag information.
#######################################################################################################
Add packages to function names in bash (#10670) (#10696) Inspired by the Google Shell Guide where they mentioned separating package names with :: I realized that this was one of the missing pieces in the bash scripts of ours. While we already had packages (in libraries folders) it's been difficult to realise which function is where. With introducing packages - equal to the library file name we are *almost* at a level of a structured language - and it's easier to find the functions if you are looking for them. Way easier in fact. Part of #10576 (cherry picked from commit cc551ba793344800d2d396c13d7fd0c8eed97352) (cherry picked from commit 2bba276f0f06a5981bdd7e4f0e7e5ca2fe84f063)
2020-09-02 22:58:37 +03:00
function breeze::flag_assume_answers_to_questions() {
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
echo "
[AIRFLOW-3611] Simplified development environment (#4932)
2019-08-27 21:39:36 +03:00
-y, --assume-yes
Assume 'yes' answer to all questions.
[AIRFLOW-5369] Adds interactivity to pre-commits (#5976) This commit adds full interactivity to pre-commits. Whenever you run pre-commit and it detects that the image should be rebuild, an interactive question will pop up instead of failing the build and asking to rebuild with REBUILD=yes This is much nicer from the user perspective. You can choose whether to: 1) Rebuild the image (which will take some time) 2) Not rebuild the image (this will use the old image with hope it's OK) 3) Quit. Answer to that question is carried across all images needed to rebuild. There is the special "build" pre-commit hook that takes care about that. Note that this interactive question cannot be asked if you run only single pre-commit hook with Dockerfile because it can run multiple processes and you can start building in parallel. This is not desired so instead we fail such builds.
2019-09-18 14:43:30 +03:00
-n, --assume-no
Assume 'no' answer to all questions.
[AIRFLOW-6838] Introduce real subcommands for Breeze (#7515) This change introduces sub-commands in breeze tool. It is much needed as we have many commands now and it was difficult to separate commands from flags. Also --help output was very long and unreadable. With this change help it is much easier to discover what breeze can do for you as well as navigate with it. Co-authored-by: Jarek Potiuk <jarek@potiuk.com> Co-authored-by: Kamil Breguła <mik-laj@users.noreply.github.com>
2020-02-25 00:31:50 +03:00
-q, --assume-quit
Assume 'quit' answer to all questions.
Help for breeze commands contain relevant flags. (#8261) * Help for breeze commands contain relevant flags. * fixup! Help for breeze commands contain relevant flags. * fixup! fixup! Help for breeze commands contain relevant flags.
2020-04-13 09:51:33 +03:00
"
}
[AIRFLOW-6838] Introduce real subcommands for Breeze (#7515) This change introduces sub-commands in breeze tool. It is much needed as we have many commands now and it was difficult to separate commands from flags. Also --help output was very long and unreadable. With this change help it is much easier to discover what breeze can do for you as well as navigate with it. Co-authored-by: Jarek Potiuk <jarek@potiuk.com> Co-authored-by: Kamil Breguła <mik-laj@users.noreply.github.com>
2020-02-25 00:31:50 +03:00
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
#######################################################################################################
#
Add comments to breeze scripts (#8797)
2020-05-10 20:55:40 +03:00
# Prints flags that are used for credential forwarding
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
#
# Outputs:
# Flag information.
#######################################################################################################
Add packages to function names in bash (#10670) (#10696) Inspired by the Google Shell Guide where they mentioned separating package names with :: I realized that this was one of the missing pieces in the bash scripts of ours. While we already had packages (in libraries folders) it's been difficult to realise which function is where. With introducing packages - equal to the library file name we are *almost* at a level of a structured language - and it's easier to find the functions if you are looking for them. Way easier in fact. Part of #10576 (cherry picked from commit cc551ba793344800d2d396c13d7fd0c8eed97352) (cherry picked from commit 2bba276f0f06a5981bdd7e4f0e7e5ca2fe84f063)
2020-09-02 22:58:37 +03:00
function breeze::flag_credentials() {
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
echo "
[AIRFLOW-6763] Make systems tests ready for backport tests (#7389) We will run system test on back-ported operators for 1.10* series of airflow and for that we need to have support for running system tests using pytest's markers and reading environment variables passed from HOST machine (to pass credentials).  This is the first step to automate system tests execution.
2020-02-21 20:25:32 +03:00
-f, --forward-credentials
[AIRFLOW-6838] Introduce real subcommands for Breeze (#7515) This change introduces sub-commands in breeze tool. It is much needed as we have many commands now and it was difficult to separate commands from flags. Also --help output was very long and unreadable. With this change help it is much easier to discover what breeze can do for you as well as navigate with it. Co-authored-by: Jarek Potiuk <jarek@potiuk.com> Co-authored-by: Kamil Breguła <mik-laj@users.noreply.github.com>
2020-02-25 00:31:50 +03:00
Forwards host credentials to docker container. Use with care as it will make
your credentials available to everything you install in Docker.
Help for breeze commands contain relevant flags. (#8261) * Help for breeze commands contain relevant flags. * fixup! Help for breeze commands contain relevant flags. * fixup! fixup! Help for breeze commands contain relevant flags.
2020-04-13 09:51:33 +03:00
"
}
[AIRFLOW-6763] Make systems tests ready for backport tests (#7389) We will run system test on back-ported operators for 1.10* series of airflow and for that we need to have support for running system tests using pytest's markers and reading environment variables passed from HOST machine (to pass credentials).  This is the first step to automate system tests execution.
2020-02-21 20:25:32 +03:00
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
#######################################################################################################
#
Add comments to breeze scripts (#8797)
2020-05-10 20:55:40 +03:00
# Prints flags that control verbosity
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
#
# Outputs:
# Flag information.
#######################################################################################################
Add packages to function names in bash (#10670) (#10696) Inspired by the Google Shell Guide where they mentioned separating package names with :: I realized that this was one of the missing pieces in the bash scripts of ours. While we already had packages (in libraries folders) it's been difficult to realise which function is where. With introducing packages - equal to the library file name we are *almost* at a level of a structured language - and it's easier to find the functions if you are looking for them. Way easier in fact. Part of #10576 (cherry picked from commit cc551ba793344800d2d396c13d7fd0c8eed97352) (cherry picked from commit 2bba276f0f06a5981bdd7e4f0e7e5ca2fe84f063)
2020-09-02 22:58:37 +03:00
function breeze::flag_verbosity() {
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
echo "
[AIRFLOW-6489] Separate integrations in tests (#7091) You can now choose which integration you want to start when you run Breeze as well as when CI tests are run. Now by default Breeze and CI runs without integrations, but you can add them via Breeze flags or by environment variables when CI is executed. We have pytest markers now that mark tests that can be run for integration, backend and runtime selected. Also we have now more test jobs - we have separate test run for all non-integration tests (with less memory used by the integrations) and separate jobs that run integration tests only (more memory used for integrations but far less number of tests to run)
2020-01-13 22:47:37 +03:00
-v, --verbose
More informative description of Breeze's --verbose flag. (#10348)
2020-08-15 21:33:02 +03:00
Show verbose information about executed docker, kind, kubectl, helm commands. Useful for
debugging - when you run breeze with --verbose flags you will be able to see the commands
executed under the hood and copy&paste them to your terminal to debug them more easily.
[AIRFLOW-6838] Introduce real subcommands for Breeze (#7515) This change introduces sub-commands in breeze tool. It is much needed as we have many commands now and it was difficult to separate commands from flags. Also --help output was very long and unreadable. With this change help it is much easier to discover what breeze can do for you as well as navigate with it. Co-authored-by: Jarek Potiuk <jarek@potiuk.com> Co-authored-by: Kamil Breguła <mik-laj@users.noreply.github.com>
2020-02-25 00:31:50 +03:00
Note that you can further increase verbosity and see all the commands executed by breeze
by running 'export VERBOSE_COMMANDS=\"true\"' before running breeze.
Help for breeze commands contain relevant flags. (#8261) * Help for breeze commands contain relevant flags. * fixup! Help for breeze commands contain relevant flags. * fixup! fixup! Help for breeze commands contain relevant flags.
2020-04-13 09:51:33 +03:00
"
}
[AIRFLOW-3611] Simplified development environment (#4932)
2019-08-27 21:39:36 +03:00
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
#######################################################################################################
#
# Prints information about help flag
#
# Outputs:
# Flag information.
#######################################################################################################
Add packages to function names in bash (#10670) (#10696) Inspired by the Google Shell Guide where they mentioned separating package names with :: I realized that this was one of the missing pieces in the bash scripts of ours. While we already had packages (in libraries folders) it's been difficult to realise which function is where. With introducing packages - equal to the library file name we are *almost* at a level of a structured language - and it's easier to find the functions if you are looking for them. Way easier in fact. Part of #10576 (cherry picked from commit cc551ba793344800d2d396c13d7fd0c8eed97352) (cherry picked from commit 2bba276f0f06a5981bdd7e4f0e7e5ca2fe84f063)
2020-09-02 22:58:37 +03:00
function breeze::flag_help() {
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
echo "
Kubernetes Cluster is started on host not in the container (#8265) Tests requiring Kubernetes Cluster are now moved out of the regular CI tests and moved to "kubernetes_tests" folder so that they can be run entirely on host without having the CI image built at all. They use production image to run the tests on KinD cluster and we add tooling to start/stop/deploy the application to the KinD cluster automatically - for both CI testing and local development. This is a pre-requisite to convert the tests to convert the tests to use the official Helm Chart and Docker images or Apache Airflow. It closes #8782
2020-06-03 21:58:38 +03:00
-h, --help
Shows detailed help message for the command specified.
"
}
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
#######################################################################################################
#
Add comments to breeze scripts (#8797)
2020-05-10 20:55:40 +03:00
# Prints flags controlling docker build process
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
#
# Global constants used:
# FORMATTED_DEFAULT_CI_EXTRAS
# FORMATTED_DEFAULT_PROD_EXTRAS
#
# Outputs:
# Flag information.
#######################################################################################################
Add packages to function names in bash (#10670) (#10696) Inspired by the Google Shell Guide where they mentioned separating package names with :: I realized that this was one of the missing pieces in the bash scripts of ours. While we already had packages (in libraries folders) it's been difficult to realise which function is where. With introducing packages - equal to the library file name we are *almost* at a level of a structured language - and it's easier to find the functions if you are looking for them. Way easier in fact. Part of #10576 (cherry picked from commit cc551ba793344800d2d396c13d7fd0c8eed97352) (cherry picked from commit 2bba276f0f06a5981bdd7e4f0e7e5ca2fe84f063)
2020-09-02 22:58:37 +03:00
function breeze::flag_build_docker_images() {
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
echo "
[AIRFLOW-6838] Introduce real subcommands for Breeze (#7515) This change introduces sub-commands in breeze tool. It is much needed as we have many commands now and it was difficult to separate commands from flags. Also --help output was very long and unreadable. With this change help it is much easier to discover what breeze can do for you as well as navigate with it. Co-authored-by: Jarek Potiuk <jarek@potiuk.com> Co-authored-by: Kamil Breguła <mik-laj@users.noreply.github.com>
2020-02-25 00:31:50 +03:00
-F, --force-build-images
[AIRFLOW-5369] Adds interactivity to pre-commits (#5976) This commit adds full interactivity to pre-commits. Whenever you run pre-commit and it detects that the image should be rebuild, an interactive question will pop up instead of failing the build and asking to rebuild with REBUILD=yes This is much nicer from the user perspective. You can choose whether to: 1) Rebuild the image (which will take some time) 2) Not rebuild the image (this will use the old image with hope it's OK) 3) Quit. Answer to that question is carried across all images needed to rebuild. There is the special "build" pre-commit hook that takes care about that. Note that this interactive question cannot be asked if you run only single pre-commit hook with Dockerfile because it can run multiple processes and you can start building in parallel. This is not desired so instead we fail such builds.
2019-09-18 14:43:30 +03:00
Forces building of the local docker images. The images are rebuilt
[AIRFLOW-3611] Simplified development environment (#4932)
2019-08-27 21:39:36 +03:00
automatically for the first time or when changes are detected in
package-related files, but you can force it using this flag.
Add Production Docker image support (#7832)
2020-04-02 20:52:11 +03:00
-P, --force-pull-images
[AIRFLOW-3611] Simplified development environment (#4932)
2019-08-27 21:39:36 +03:00
Forces pulling of images from DockerHub before building to populate cache. The
images are pulled by default only for the first time you run the
environment, later the locally build images are used as cache.
Add Production Docker image support (#7832)
2020-04-02 20:52:11 +03:00
-E, --extras
Extras to pass to build images The default are different for CI and production images:
CI image:
${FORMATTED_DEFAULT_CI_EXTRAS}
Production image:
${FORMATTED_DEFAULT_PROD_EXTRAS}
Additional python extras and deps can be set in breeze (#9035) Closes #8604 Closes #8866
2020-05-27 18:09:11 +03:00
--additional-extras
Additional extras to pass to build images The default is no additional extras.
--additional-python-deps
Additional python dependencies to use when building the images.
Additional apt dependencies options in breeze (#9231)
2020-06-11 19:53:26 +03:00
--additional-dev-deps
Additional apt dev dependencies to use when building the images.
--additional-runtime-deps
Additional apt runtime dependencies to use when building the images.
Optionally disables PIP cache from GitHub during the build (#11173) This is first step of implementing the corporate-environment friendly way of building images, where in the corporate environment, this might not be possible to install the packages using the GitHub cache initially. Part of #11171
2020-09-27 19:00:03 +03:00
--disable-pip-cache
Disables GitHub PIP cache during the build. Useful if github is not reachable during build.
Conditional MySQL Client installation (#11174) This is the second step of making the Production Docker Image more corporate-environment friendly, by making MySQL client installation optional. Instaling MySQL Client on Debian requires to reach out to oracle deb repositories which might not be approved by security teams when you build the images. Also not everyone needs MySQL client or might want to install their own MySQL client or MariaDB client - from their own repositories. This change makes the installation step separated out to script (with prod/dev installation option). The prod/dev separation is needed because MySQL needs to be installed with dev libraries in the "Build" segment of the image (requiring build essentials etc.) but in "Final" segment of the image only runtime libraries are needed. Part of #11171 Depends on #11173.
2020-09-27 19:56:58 +03:00
--disable-mysql-client-installation
Disables installation of the mysql client which might be problematic if you are building
image in controlled environment. Only valid for production image.
[AIRFLOW-6838] Introduce real subcommands for Breeze (#7515) This change introduces sub-commands in breeze tool. It is much needed as we have many commands now and it was difficult to separate commands from flags. Also --help output was very long and unreadable. With this change help it is much easier to discover what breeze can do for you as well as navigate with it. Co-authored-by: Jarek Potiuk <jarek@potiuk.com> Co-authored-by: Kamil Breguła <mik-laj@users.noreply.github.com>
2020-02-25 00:31:50 +03:00
-C, --force-clean-images
[AIRFLOW-6489] Separate integrations in tests (#7091) You can now choose which integration you want to start when you run Breeze as well as when CI tests are run. Now by default Breeze and CI runs without integrations, but you can add them via Breeze flags or by environment variables when CI is executed. We have pytest markers now that mark tests that can be run for integration, backend and runtime selected. Also we have now more test jobs - we have separate test run for all non-integration tests (with less memory used by the integrations) and separate jobs that run integration tests only (more memory used for integrations but far less number of tests to run)
2020-01-13 22:47:37 +03:00
Force build images with cache disabled. This will remove the pulled or build images
[AIRFLOW-5704] Improve Kind Kubernetes scripts for local testing (#6516) * Fixed problem that Kubernetes tests were testing latest master rather than what came from the local sources. * Kind (Kubernetes in Dcocker) is run in the same Docker as Breeze env * Moved Kubernetes scripts to 'in_container' dir where they belong now * Kubernetes cluster is reused until it is stopped * Kubernetes image is build from image in docker already + mounted sources * Kubectl version name is corrected in the Dockerfile * KUBERNETES_VERSION can now be used to select Kubernetes version * Running kubernetes scripts is now easy in Breeze * We can start/recreate/stop cluster using --<ACTION>-kind-cluster * Instructions on how to run Kubernetes tests are updated * The old "bare" environment is replaced by --no-deps switch
2020-01-11 18:25:19 +03:00
and start building images from scratch. This might take a long time.
CI Images are now pre-build and stored in registry (#10368) * CI Images are now pre-build and stored in registry With this change we utilise the latest pull_request_target event type from Github Actions and we are building the CI image only once (per version) for the entire run. This safes from 2 to 10 minutes per job (!) depending on how much of the Docker image needs to be rebuilt. It works in the way that the image is built only in the build-or-wait step. In case of direct push run or scheduled runs, the build-or-wait step builds and pushes to the GitHub registry the CI image. In case of the pull_request runs, the build-and-wait step waits until separate build-ci-image.yml workflow builds and pushes the image and it will only move forward once the image is ready. This has numerous advantages: 1) Each job that requires CI image is much faster because instead of pulling + rebuilding the image it only pulls the image that was build once. This saves around 2 minutes per job in regular builds but in case of python patch level updates, or adding new requirements it can save up to 10 minutes per job (!) 2) While the images are buing rebuilt we only block one job waiting for all the images. The tests will start running in parallell only when all images are ready, so we are not blocking other runs from running. 3) Whole run uses THE SAME image. Previously we could have some variations because the images were built at different times and potentially releases of dependencies in-between several jobs could make different jobs in the same run use slightly different image. This is not happening any more. 4) Also when we push image to github or dockerhub we push the very same image that was built and tested. Previously it could happen that the image pushed was slightly different than the one that was used for testing (for the same reason) 5) Similar case is with the production images. We are now building and pushing consistently the same images accross the board. 6) Documentation building is split into two parallel jobs docs building and spell checking - decreases elapsed time for the docs build. 7) Last but not least - we keep the history of al the images - those images contain SHA of the commit. This means that we can simply download and run the image locally to reproduce any problem that anyone had in their PR (!). This is super useful to be able to help others to test their problems. * fixup! CI Images are now pre-build and stored in registry * fixup! fixup! CI Images are now pre-build and stored in registry * fixup! fixup! fixup! CI Images are now pre-build and stored in registry * fixup! fixup! fixup! CI Images are now pre-build and stored in registry
2020-08-20 10:57:07 +03:00
-r, --skip-rebuild-check
Skips checking image for rebuilds. It will use whatever image is available locally/pulled.
More sensible docker caching strategy for Prod images (#9547) Local caching is now default strategy when building the Production image. You can still change it to pulled - similar to CI builds by providing the right build flag and this is what is used in CI by default. The flags in Breeze are now updated to be more eplanatory and friendly (build-cache-*) and a flag for "disabled" cache option is added as well. Also the Dockerfile and Dockerfile.ci files are not needed any more in the docker context. They used to be needed when we built the Kubernetes image in the container, but since we are now using production image directly - we do not need them any nmore. Combining setting the default strategy to local and removing the Dockerfile from the context has the nice effect that you can iterate much faster on the Production image without triggering rebuilds of half of the docker image as soon as the Dockerfile changes.
2020-06-28 18:38:17 +03:00
-L, --build-cache-local
[AIRFLOW-6838] Introduce real subcommands for Breeze (#7515) This change introduces sub-commands in breeze tool. It is much needed as we have many commands now and it was difficult to separate commands from flags. Also --help output was very long and unreadable. With this change help it is much easier to discover what breeze can do for you as well as navigate with it. Co-authored-by: Jarek Potiuk <jarek@potiuk.com> Co-authored-by: Kamil Breguła <mik-laj@users.noreply.github.com>
2020-02-25 00:31:50 +03:00
Uses local cache to build images. No pulled images will be used, but results of local
More sensible docker caching strategy for Prod images (#9547) Local caching is now default strategy when building the Production image. You can still change it to pulled - similar to CI builds by providing the right build flag and this is what is used in CI by default. The flags in Breeze are now updated to be more eplanatory and friendly (build-cache-*) and a flag for "disabled" cache option is added as well. Also the Dockerfile and Dockerfile.ci files are not needed any more in the docker context. They used to be needed when we built the Kubernetes image in the container, but since we are now using production image directly - we do not need them any nmore. Combining setting the default strategy to local and removing the Dockerfile from the context has the nice effect that you can iterate much faster on the Production image without triggering rebuilds of half of the docker image as soon as the Dockerfile changes.
2020-06-28 18:38:17 +03:00
builds in the Docker cache are used instead. This will take longer than when the pulled
cache is used for the first time, but subsequent '--build-cache-local' builds will be
faster as they will use mostly the locally build cache.
This is default strategy used by the Production image builds.
-U, --build-cache-pulled
Uses images pulled from registry (either DockerHub or GitHub depending on
--github-registry flag) to build images. The pulled images will be used as cache.
Those builds are usually faster than when ''--build-cache-local'' with the exception if
the registry images are not yet updated. The DockerHub images are updated nightly and the
GitHub images are updated after merges to master so it might be that the images are still
outdated vs. the latest version of the Dockerfiles you are using. In this case, the
''--build-cache-local'' might be faster, especially if you iterate and change the
Dockerfiles yourself.
This is default strategy used by the CI image builds.
-X, --build-cache-disabled
Disables cache during docker builds. This is useful if you want to make sure you want to
rebuild everything from scratch.
This strategy is used by default for both Production and CI images for the scheduled
(nightly) builds in CI.
Help for breeze commands contain relevant flags. (#8261) * Help for breeze commands contain relevant flags. * fixup! Help for breeze commands contain relevant flags. * fixup! fixup! Help for breeze commands contain relevant flags.
2020-04-13 09:51:33 +03:00
"
}
[AIRFLOW-5704] Improve Kind Kubernetes scripts for local testing (#6516) * Fixed problem that Kubernetes tests were testing latest master rather than what came from the local sources. * Kind (Kubernetes in Dcocker) is run in the same Docker as Breeze env * Moved Kubernetes scripts to 'in_container' dir where they belong now * Kubernetes cluster is reused until it is stopped * Kubernetes image is build from image in docker already + mounted sources * Kubectl version name is corrected in the Dockerfile * KUBERNETES_VERSION can now be used to select Kubernetes version * Running kubernetes scripts is now easy in Breeze * We can start/recreate/stop cluster using --<ACTION>-kind-cluster * Instructions on how to run Kubernetes tests are updated * The old "bare" environment is replaced by --no-deps switch
2020-01-11 18:25:19 +03:00
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
#######################################################################################################
#
You can push with Breeze as separate command and to cache (#8976) Breeze had --push-images switch to also push images to repo but it was often needed to build and push images separately. We have now a possibility to push an already built image with separate push-image command instead and also you can choose to push to cache registry in GitHub rather than to DockerHub with --registry-cache switch.
2020-06-02 11:24:47 +03:00
# Prints flags controlling docker pull and push process
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
#
# Global constants used:
Make breeeze-complete Google Shell Guide compatible (#10708) Also added unit tests for breeze-complete Part of #10576
2020-09-14 11:21:09 +03:00
# _breeze_default_dockerhub_user
# _breeze_default_dockerhub_repo
# _breeze_default_github_repository
# _breeze_default_github_image_id
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
#
# Outputs:
# Flag information.
#######################################################################################################
Add packages to function names in bash (#10670) (#10696) Inspired by the Google Shell Guide where they mentioned separating package names with :: I realized that this was one of the missing pieces in the bash scripts of ours. While we already had packages (in libraries folders) it's been difficult to realise which function is where. With introducing packages - equal to the library file name we are *almost* at a level of a structured language - and it's easier to find the functions if you are looking for them. Way easier in fact. Part of #10576 (cherry picked from commit cc551ba793344800d2d396c13d7fd0c8eed97352) (cherry picked from commit 2bba276f0f06a5981bdd7e4f0e7e5ca2fe84f063)
2020-09-02 22:58:37 +03:00
function breeze::flag_pull_push_docker_images() {
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
echo "
[AIRFLOW-6489] Separate integrations in tests (#7091) You can now choose which integration you want to start when you run Breeze as well as when CI tests are run. Now by default Breeze and CI runs without integrations, but you can add them via Breeze flags or by environment variables when CI is executed. We have pytest markers now that mark tests that can be run for integration, backend and runtime selected. Also we have now more test jobs - we have separate test run for all non-integration tests (with less memory used by the integrations) and separate jobs that run integration tests only (more memory used for integrations but far less number of tests to run)
2020-01-13 22:47:37 +03:00
-D, --dockerhub-user
Make breeeze-complete Google Shell Guide compatible (#10708) Also added unit tests for breeze-complete Part of #10576
2020-09-14 11:21:09 +03:00
DockerHub user used to pull, push and build images. Default: ${_breeze_default_dockerhub_user:=}.
[AIRFLOW-6489] Separate integrations in tests (#7091) You can now choose which integration you want to start when you run Breeze as well as when CI tests are run. Now by default Breeze and CI runs without integrations, but you can add them via Breeze flags or by environment variables when CI is executed. We have pytest markers now that mark tests that can be run for integration, backend and runtime selected. Also we have now more test jobs - we have separate test run for all non-integration tests (with less memory used by the integrations) and separate jobs that run integration tests only (more memory used for integrations but far less number of tests to run)
2020-01-13 22:47:37 +03:00
-H, --dockerhub-repo
Make breeeze-complete Google Shell Guide compatible (#10708) Also added unit tests for breeze-complete Part of #10576
2020-09-14 11:21:09 +03:00
DockerHub repository used to pull, push, build images. Default: ${_breeze_default_dockerhub_repo:=}.
You can push with Breeze as separate command and to cache (#8976) Breeze had --push-images switch to also push images to repo but it was often needed to build and push images separately. We have now a possibility to push an already built image with separate push-image command instead and also you can choose to push to cache registry in GitHub rather than to DockerHub with --registry-cache switch.
2020-06-02 11:24:47 +03:00
Improve production image iteration speed (#9162) For a long time the way how entrypoint worked in ci scripts was wrong. The way it worked was convoluted and short of black magic. This did not allow to pass multiple test targets and required separate execute command scripts in Breeze. This is all now straightened out and both production and CI image are always using the right entrypoint by default and we can simply pass parameters to the image as usual without escaping strings. This also allowed to remove some breeze commands and change names of several flags in Breeze to make them more meaningful. Both CI and PROD image have now embedded scripts for log cleaning. History of image releases is added for 1.10.10-* alpha quality images.
2020-06-16 13:36:46 +03:00
-c, --github-registry
If GitHub registry is enabled, pulls and pushes are done from the GitHub registry not
Update github flags for Breeze (#10384) Part of #10368
2020-08-19 01:53:04 +03:00
DockerHub. You need to be logged in to the registry in order to be able to pull/push from
Improve production image iteration speed (#9162) For a long time the way how entrypoint worked in ci scripts was wrong. The way it worked was convoluted and short of black magic. This did not allow to pass multiple test targets and required separate execute command scripts in Breeze. This is all now straightened out and both production and CI image are always using the right entrypoint by default and we can simply pass parameters to the image as usual without escaping strings. This also allowed to remove some breeze commands and change names of several flags in Breeze to make them more meaningful. Both CI and PROD image have now embedded scripts for log cleaning. History of image releases is added for 1.10.10-* alpha quality images.
2020-06-16 13:36:46 +03:00
and you need to be committer to push to Apache Airflow' GitHub registry.
You can push with Breeze as separate command and to cache (#8976) Breeze had --push-images switch to also push images to repo but it was often needed to build and push images separately. We have now a possibility to push an already built image with separate push-image command instead and also you can choose to push to cache registry in GitHub rather than to DockerHub with --registry-cache switch.
2020-06-02 11:24:47 +03:00
Update github flags for Breeze (#10384) Part of #10368
2020-08-19 01:53:04 +03:00
-g, --github-repository
GitHub repository used to pull, push images when cache is used.
Make breeeze-complete Google Shell Guide compatible (#10708) Also added unit tests for breeze-complete Part of #10576
2020-09-14 11:21:09 +03:00
Default: ${_breeze_default_github_repository:=}.
Update github flags for Breeze (#10384) Part of #10368
2020-08-19 01:53:04 +03:00
If you use this flag, automatically --github-registry flag is enabled.
-s, --github-image-id <COMMIT_SHA>|<RUN_ID>
<RUN_ID> or <COMMIT_SHA> of the image. Images in GitHub registry are stored with those
to be able to easily find the image for particular CI runs. Once you know the
<RUN_ID> or <COMMIT_SHA>, you can specify it in github-image-id flag and Breeze will
automatically pull and use that image so that you can easily reproduce a problem
that occurred in CI.
If you use this flag, automatically --github-registry is enabled.
You can push with Breeze as separate command and to cache (#8976) Breeze had --push-images switch to also push images to repo but it was often needed to build and push images separately. We have now a possibility to push an already built image with separate push-image command instead and also you can choose to push to cache registry in GitHub rather than to DockerHub with --registry-cache switch.
2020-06-02 11:24:47 +03:00
Make breeeze-complete Google Shell Guide compatible (#10708) Also added unit tests for breeze-complete Part of #10576
2020-09-14 11:21:09 +03:00
Default: ${_breeze_default_github_image_id:=}.
You can push with Breeze as separate command and to cache (#8976) Breeze had --push-images switch to also push images to repo but it was often needed to build and push images separately. We have now a possibility to push an already built image with separate push-image command instead and also you can choose to push to cache registry in GitHub rather than to DockerHub with --registry-cache switch.
2020-06-02 11:24:47 +03:00
Help for breeze commands contain relevant flags. (#8261) * Help for breeze commands contain relevant flags. * fixup! Help for breeze commands contain relevant flags. * fixup! fixup! Help for breeze commands contain relevant flags.
2020-04-13 09:51:33 +03:00
"
}
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
#######################################################################################################
#
Added automated release notes generation for backport operators (#8807) We have now mechanism to keep release notes updated for the backport operators in an automated way. It really nicely generates all the necessary information: * summary of requirements for each backport package * list of dependencies (including extras to install them) when package depends on other providers packages * table of new hooks/operators/sensors/protocols/secrets * table of moved hooks/operators/sensors/protocols/secrets with information where they were moved from * changelog of all the changes to the provider package (this will be automatically updated with incremental changelog whenever we decide to release separate packages. The system is fully automated - we will be able to produce release notes automatically (per-package) whenever we decide to release new version of the package in the future.
2020-05-15 20:00:15 +03:00
# Prints flags that control version of generated packages
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
#
# Outputs:
# Flag information.
#######################################################################################################
Add packages to function names in bash (#10670) (#10696) Inspired by the Google Shell Guide where they mentioned separating package names with :: I realized that this was one of the missing pieces in the bash scripts of ours. While we already had packages (in libraries folders) it's been difficult to realise which function is where. With introducing packages - equal to the library file name we are *almost* at a level of a structured language - and it's easier to find the functions if you are looking for them. Way easier in fact. Part of #10576 (cherry picked from commit cc551ba793344800d2d396c13d7fd0c8eed97352) (cherry picked from commit 2bba276f0f06a5981bdd7e4f0e7e5ca2fe84f063)
2020-09-02 22:58:37 +03:00
function breeze::flag_version_suffix() {
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
echo "
Release candidate 2 for backport packages 2020.05.20 (#8898) Release candidate 2 for backport packages 2020.05.20
2020-05-19 15:17:22 +03:00
-S, --version-suffix-for-pypi
Adds optional suffix to the version in the generated backport package. It can be used
to generate rc1/rc2 ... versions of the packages to be uploaded to PyPI.
-N, --version-suffix-for-svn
Adds optional suffix to the generated names of package. It can be used to generate
rc1/rc2 ... versions of the packages to be uploaded to SVN.
Added automated release notes generation for backport operators (#8807) We have now mechanism to keep release notes updated for the backport operators in an automated way. It really nicely generates all the necessary information: * summary of requirements for each backport package * list of dependencies (including extras to install them) when package depends on other providers packages * table of new hooks/operators/sensors/protocols/secrets * table of moved hooks/operators/sensors/protocols/secrets with information where they were moved from * changelog of all the changes to the provider package (this will be automatically updated with incremental changelog whenever we decide to release separate packages. The system is fully automated - we will be able to produce release notes automatically (per-package) whenever we decide to release new version of the package in the future.
2020-05-15 20:00:15 +03:00
"
}
New Breeze command start-airflow, it replaces the previous flag (#11157)
2020-09-27 19:31:50 +03:00
#####################################################################################################
#
# Prints flags that control how Airflow should be populated with the command start-airflow
#
# Outputs:
# Flag information.
#######################################################################################################
function breeze::flag_start_airflow() {
echo "
--load-example-dags
Include Airflow example dags.
--load-default-connections
Include Airflow Default Connections.
"
}
Added automated release notes generation for backport operators (#8807) We have now mechanism to keep release notes updated for the backport operators in an automated way. It really nicely generates all the necessary information: * summary of requirements for each backport package * list of dependencies (including extras to install them) when package depends on other providers packages * table of new hooks/operators/sensors/protocols/secrets * table of moved hooks/operators/sensors/protocols/secrets with information where they were moved from * changelog of all the changes to the provider package (this will be automatically updated with incremental changelog whenever we decide to release separate packages. The system is fully automated - we will be able to produce release notes automatically (per-package) whenever we decide to release new version of the package in the future.
2020-05-15 20:00:15 +03:00
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
#######################################################################################################
#
Add comments to breeze scripts (#8797)
2020-05-10 20:55:40 +03:00
# Prints all flags
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
#
# Outputs:
# Flag information.
#######################################################################################################
Add packages to function names in bash (#10670) (#10696) Inspired by the Google Shell Guide where they mentioned separating package names with :: I realized that this was one of the missing pieces in the bash scripts of ours. While we already had packages (in libraries folders) it's been difficult to realise which function is where. With introducing packages - equal to the library file name we are *almost* at a level of a structured language - and it's easier to find the functions if you are looking for them. Way easier in fact. Part of #10576 (cherry picked from commit cc551ba793344800d2d396c13d7fd0c8eed97352) (cherry picked from commit 2bba276f0f06a5981bdd7e4f0e7e5ca2fe84f063)
2020-09-02 22:58:37 +03:00
function breeze::flags() {
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
echo "
Add packages to function names in bash (#10670) (#10696) Inspired by the Google Shell Guide where they mentioned separating package names with :: I realized that this was one of the missing pieces in the bash scripts of ours. While we already had packages (in libraries folders) it's been difficult to realise which function is where. With introducing packages - equal to the library file name we are *almost* at a level of a structured language - and it's easier to find the functions if you are looking for them. Way easier in fact. Part of #10576 (cherry picked from commit cc551ba793344800d2d396c13d7fd0c8eed97352) (cherry picked from commit 2bba276f0f06a5981bdd7e4f0e7e5ca2fe84f063)
2020-09-02 22:58:37 +03:00
$(breeze::print_line)
Help for breeze commands contain relevant flags. (#8261) * Help for breeze commands contain relevant flags. * fixup! Help for breeze commands contain relevant flags. * fixup! fixup! Help for breeze commands contain relevant flags.
2020-04-13 09:51:33 +03:00
Summary of all flags supported by Breeze:
Add packages to function names in bash (#10670) (#10696) Inspired by the Google Shell Guide where they mentioned separating package names with :: I realized that this was one of the missing pieces in the bash scripts of ours. While we already had packages (in libraries folders) it's been difficult to realise which function is where. With introducing packages - equal to the library file name we are *almost* at a level of a structured language - and it's easier to find the functions if you are looking for them. Way easier in fact. Part of #10576 (cherry picked from commit cc551ba793344800d2d396c13d7fd0c8eed97352) (cherry picked from commit 2bba276f0f06a5981bdd7e4f0e7e5ca2fe84f063)
2020-09-02 22:58:37 +03:00
$(breeze::print_star_line)
Help for breeze commands contain relevant flags. (#8261) * Help for breeze commands contain relevant flags. * fixup! Help for breeze commands contain relevant flags. * fixup! fixup! Help for breeze commands contain relevant flags.
2020-04-13 09:51:33 +03:00
Choose Airflow variant
Add packages to function names in bash (#10670) (#10696) Inspired by the Google Shell Guide where they mentioned separating package names with :: I realized that this was one of the missing pieces in the bash scripts of ours. While we already had packages (in libraries folders) it's been difficult to realise which function is where. With introducing packages - equal to the library file name we are *almost* at a level of a structured language - and it's easier to find the functions if you are looking for them. Way easier in fact. Part of #10576 (cherry picked from commit cc551ba793344800d2d396c13d7fd0c8eed97352) (cherry picked from commit 2bba276f0f06a5981bdd7e4f0e7e5ca2fe84f063)
2020-09-02 22:58:37 +03:00
$(breeze::flag_airflow_variants)
Help for breeze commands contain relevant flags. (#8261) * Help for breeze commands contain relevant flags. * fixup! Help for breeze commands contain relevant flags. * fixup! fixup! Help for breeze commands contain relevant flags.
2020-04-13 09:51:33 +03:00
Add packages to function names in bash (#10670) (#10696) Inspired by the Google Shell Guide where they mentioned separating package names with :: I realized that this was one of the missing pieces in the bash scripts of ours. While we already had packages (in libraries folders) it's been difficult to realise which function is where. With introducing packages - equal to the library file name we are *almost* at a level of a structured language - and it's easier to find the functions if you are looking for them. Way easier in fact. Part of #10576 (cherry picked from commit cc551ba793344800d2d396c13d7fd0c8eed97352) (cherry picked from commit 2bba276f0f06a5981bdd7e4f0e7e5ca2fe84f063)
2020-09-02 22:58:37 +03:00
$(breeze::print_star_line)
Help for breeze commands contain relevant flags. (#8261) * Help for breeze commands contain relevant flags. * fixup! Help for breeze commands contain relevant flags. * fixup! fixup! Help for breeze commands contain relevant flags.
2020-04-13 09:51:33 +03:00
Choose backend to run for Airflow
Add packages to function names in bash (#10670) (#10696) Inspired by the Google Shell Guide where they mentioned separating package names with :: I realized that this was one of the missing pieces in the bash scripts of ours. While we already had packages (in libraries folders) it's been difficult to realise which function is where. With introducing packages - equal to the library file name we are *almost* at a level of a structured language - and it's easier to find the functions if you are looking for them. Way easier in fact. Part of #10576 (cherry picked from commit cc551ba793344800d2d396c13d7fd0c8eed97352) (cherry picked from commit 2bba276f0f06a5981bdd7e4f0e7e5ca2fe84f063)
2020-09-02 22:58:37 +03:00
$(breeze::flag_backend_variants)
Help for breeze commands contain relevant flags. (#8261) * Help for breeze commands contain relevant flags. * fixup! Help for breeze commands contain relevant flags. * fixup! fixup! Help for breeze commands contain relevant flags.
2020-04-13 09:51:33 +03:00
Add packages to function names in bash (#10670) (#10696) Inspired by the Google Shell Guide where they mentioned separating package names with :: I realized that this was one of the missing pieces in the bash scripts of ours. While we already had packages (in libraries folders) it's been difficult to realise which function is where. With introducing packages - equal to the library file name we are *almost* at a level of a structured language - and it's easier to find the functions if you are looking for them. Way easier in fact. Part of #10576 (cherry picked from commit cc551ba793344800d2d396c13d7fd0c8eed97352) (cherry picked from commit 2bba276f0f06a5981bdd7e4f0e7e5ca2fe84f063)
2020-09-02 22:58:37 +03:00
$(breeze::print_star_line)
Help for breeze commands contain relevant flags. (#8261) * Help for breeze commands contain relevant flags. * fixup! Help for breeze commands contain relevant flags. * fixup! fixup! Help for breeze commands contain relevant flags.
2020-04-13 09:51:33 +03:00
Enable production image
Add packages to function names in bash (#10670) (#10696) Inspired by the Google Shell Guide where they mentioned separating package names with :: I realized that this was one of the missing pieces in the bash scripts of ours. While we already had packages (in libraries folders) it's been difficult to realise which function is where. With introducing packages - equal to the library file name we are *almost* at a level of a structured language - and it's easier to find the functions if you are looking for them. Way easier in fact. Part of #10576 (cherry picked from commit cc551ba793344800d2d396c13d7fd0c8eed97352) (cherry picked from commit 2bba276f0f06a5981bdd7e4f0e7e5ca2fe84f063)
2020-09-02 22:58:37 +03:00
$(breeze::flag_production_image)
Help for breeze commands contain relevant flags. (#8261) * Help for breeze commands contain relevant flags. * fixup! Help for breeze commands contain relevant flags. * fixup! fixup! Help for breeze commands contain relevant flags.
2020-04-13 09:51:33 +03:00
Add packages to function names in bash (#10670) (#10696) Inspired by the Google Shell Guide where they mentioned separating package names with :: I realized that this was one of the missing pieces in the bash scripts of ours. While we already had packages (in libraries folders) it's been difficult to realise which function is where. With introducing packages - equal to the library file name we are *almost* at a level of a structured language - and it's easier to find the functions if you are looking for them. Way easier in fact. Part of #10576 (cherry picked from commit cc551ba793344800d2d396c13d7fd0c8eed97352) (cherry picked from commit 2bba276f0f06a5981bdd7e4f0e7e5ca2fe84f063)
2020-09-02 22:58:37 +03:00
$(breeze::print_star_line)
Help for breeze commands contain relevant flags. (#8261) * Help for breeze commands contain relevant flags. * fixup! Help for breeze commands contain relevant flags. * fixup! fixup! Help for breeze commands contain relevant flags.
2020-04-13 09:51:33 +03:00
Additional actions executed while entering breeze
Add packages to function names in bash (#10670) (#10696) Inspired by the Google Shell Guide where they mentioned separating package names with :: I realized that this was one of the missing pieces in the bash scripts of ours. While we already had packages (in libraries folders) it's been difficult to realise which function is where. With introducing packages - equal to the library file name we are *almost* at a level of a structured language - and it's easier to find the functions if you are looking for them. Way easier in fact. Part of #10576 (cherry picked from commit cc551ba793344800d2d396c13d7fd0c8eed97352) (cherry picked from commit 2bba276f0f06a5981bdd7e4f0e7e5ca2fe84f063)
2020-09-02 22:58:37 +03:00
$(breeze::flag_breeze_actions)
Help for breeze commands contain relevant flags. (#8261) * Help for breeze commands contain relevant flags. * fixup! Help for breeze commands contain relevant flags. * fixup! fixup! Help for breeze commands contain relevant flags.
2020-04-13 09:51:33 +03:00
New Breeze command start-airflow, it replaces the previous flag (#11157)
2020-09-27 19:31:50 +03:00
$(breeze::print_star_line)
Additional actions executed starting Airflow
$(breeze::flag_start_airflow)
Add packages to function names in bash (#10670) (#10696) Inspired by the Google Shell Guide where they mentioned separating package names with :: I realized that this was one of the missing pieces in the bash scripts of ours. While we already had packages (in libraries folders) it's been difficult to realise which function is where. With introducing packages - equal to the library file name we are *almost* at a level of a structured language - and it's easier to find the functions if you are looking for them. Way easier in fact. Part of #10576 (cherry picked from commit cc551ba793344800d2d396c13d7fd0c8eed97352) (cherry picked from commit 2bba276f0f06a5981bdd7e4f0e7e5ca2fe84f063)
2020-09-02 22:58:37 +03:00
$(breeze::print_star_line)
Kubernetes Cluster is started on host not in the container (#8265) Tests requiring Kubernetes Cluster are now moved out of the regular CI tests and moved to "kubernetes_tests" folder so that they can be run entirely on host without having the CI image built at all. They use production image to run the tests on KinD cluster and we add tooling to start/stop/deploy the application to the KinD cluster automatically - for both CI testing and local development. This is a pre-requisite to convert the tests to convert the tests to use the official Helm Chart and Docker images or Apache Airflow. It closes #8782
2020-06-03 21:58:38 +03:00
Kind kubernetes and Kubernetes tests configuration(optional)
Add packages to function names in bash (#10670) (#10696) Inspired by the Google Shell Guide where they mentioned separating package names with :: I realized that this was one of the missing pieces in the bash scripts of ours. While we already had packages (in libraries folders) it's been difficult to realise which function is where. With introducing packages - equal to the library file name we are *almost* at a level of a structured language - and it's easier to find the functions if you are looking for them. Way easier in fact. Part of #10576 (cherry picked from commit cc551ba793344800d2d396c13d7fd0c8eed97352) (cherry picked from commit 2bba276f0f06a5981bdd7e4f0e7e5ca2fe84f063)
2020-09-02 22:58:37 +03:00
$(breeze::flag_kubernetes_configuration)
Help for breeze commands contain relevant flags. (#8261) * Help for breeze commands contain relevant flags. * fixup! Help for breeze commands contain relevant flags. * fixup! fixup! Help for breeze commands contain relevant flags.
2020-04-13 09:51:33 +03:00
Add packages to function names in bash (#10670) (#10696) Inspired by the Google Shell Guide where they mentioned separating package names with :: I realized that this was one of the missing pieces in the bash scripts of ours. While we already had packages (in libraries folders) it's been difficult to realise which function is where. With introducing packages - equal to the library file name we are *almost* at a level of a structured language - and it's easier to find the functions if you are looking for them. Way easier in fact. Part of #10576 (cherry picked from commit cc551ba793344800d2d396c13d7fd0c8eed97352) (cherry picked from commit 2bba276f0f06a5981bdd7e4f0e7e5ca2fe84f063)
2020-09-02 22:58:37 +03:00
$(breeze::print_star_line)
Help for breeze commands contain relevant flags. (#8261) * Help for breeze commands contain relevant flags. * fixup! Help for breeze commands contain relevant flags. * fixup! fixup! Help for breeze commands contain relevant flags.
2020-04-13 09:51:33 +03:00
Manage mounting local files
Add packages to function names in bash (#10670) (#10696) Inspired by the Google Shell Guide where they mentioned separating package names with :: I realized that this was one of the missing pieces in the bash scripts of ours. While we already had packages (in libraries folders) it's been difficult to realise which function is where. With introducing packages - equal to the library file name we are *almost* at a level of a structured language - and it's easier to find the functions if you are looking for them. Way easier in fact. Part of #10576 (cherry picked from commit cc551ba793344800d2d396c13d7fd0c8eed97352) (cherry picked from commit 2bba276f0f06a5981bdd7e4f0e7e5ca2fe84f063)
2020-09-02 22:58:37 +03:00
$(breeze::flag_local_file_mounting)
Help for breeze commands contain relevant flags. (#8261) * Help for breeze commands contain relevant flags. * fixup! Help for breeze commands contain relevant flags. * fixup! fixup! Help for breeze commands contain relevant flags.
2020-04-13 09:51:33 +03:00
Add packages to function names in bash (#10670) (#10696) Inspired by the Google Shell Guide where they mentioned separating package names with :: I realized that this was one of the missing pieces in the bash scripts of ours. While we already had packages (in libraries folders) it's been difficult to realise which function is where. With introducing packages - equal to the library file name we are *almost* at a level of a structured language - and it's easier to find the functions if you are looking for them. Way easier in fact. Part of #10576 (cherry picked from commit cc551ba793344800d2d396c13d7fd0c8eed97352) (cherry picked from commit 2bba276f0f06a5981bdd7e4f0e7e5ca2fe84f063)
2020-09-02 22:58:37 +03:00
$(breeze::print_star_line)
Help for breeze commands contain relevant flags. (#8261) * Help for breeze commands contain relevant flags. * fixup! Help for breeze commands contain relevant flags. * fixup! fixup! Help for breeze commands contain relevant flags.
2020-04-13 09:51:33 +03:00
Assume answers to questions
Add packages to function names in bash (#10670) (#10696) Inspired by the Google Shell Guide where they mentioned separating package names with :: I realized that this was one of the missing pieces in the bash scripts of ours. While we already had packages (in libraries folders) it's been difficult to realise which function is where. With introducing packages - equal to the library file name we are *almost* at a level of a structured language - and it's easier to find the functions if you are looking for them. Way easier in fact. Part of #10576 (cherry picked from commit cc551ba793344800d2d396c13d7fd0c8eed97352) (cherry picked from commit 2bba276f0f06a5981bdd7e4f0e7e5ca2fe84f063)
2020-09-02 22:58:37 +03:00
$(breeze::flag_assume_answers_to_questions)
Help for breeze commands contain relevant flags. (#8261) * Help for breeze commands contain relevant flags. * fixup! Help for breeze commands contain relevant flags. * fixup! fixup! Help for breeze commands contain relevant flags.
2020-04-13 09:51:33 +03:00
Add packages to function names in bash (#10670) (#10696) Inspired by the Google Shell Guide where they mentioned separating package names with :: I realized that this was one of the missing pieces in the bash scripts of ours. While we already had packages (in libraries folders) it's been difficult to realise which function is where. With introducing packages - equal to the library file name we are *almost* at a level of a structured language - and it's easier to find the functions if you are looking for them. Way easier in fact. Part of #10576 (cherry picked from commit cc551ba793344800d2d396c13d7fd0c8eed97352) (cherry picked from commit 2bba276f0f06a5981bdd7e4f0e7e5ca2fe84f063)
2020-09-02 22:58:37 +03:00
$(breeze::print_star_line)
Help for breeze commands contain relevant flags. (#8261) * Help for breeze commands contain relevant flags. * fixup! Help for breeze commands contain relevant flags. * fixup! fixup! Help for breeze commands contain relevant flags.
2020-04-13 09:51:33 +03:00
Choose different Airflow version to install or run
Add packages to function names in bash (#10670) (#10696) Inspired by the Google Shell Guide where they mentioned separating package names with :: I realized that this was one of the missing pieces in the bash scripts of ours. While we already had packages (in libraries folders) it's been difficult to realise which function is where. With introducing packages - equal to the library file name we are *almost* at a level of a structured language - and it's easier to find the functions if you are looking for them. Way easier in fact. Part of #10576 (cherry picked from commit cc551ba793344800d2d396c13d7fd0c8eed97352) (cherry picked from commit 2bba276f0f06a5981bdd7e4f0e7e5ca2fe84f063)
2020-09-02 22:58:37 +03:00
$(breeze::flag_choose_different_airflow_version)
Help for breeze commands contain relevant flags. (#8261) * Help for breeze commands contain relevant flags. * fixup! Help for breeze commands contain relevant flags. * fixup! fixup! Help for breeze commands contain relevant flags.
2020-04-13 09:51:33 +03:00
Add packages to function names in bash (#10670) (#10696) Inspired by the Google Shell Guide where they mentioned separating package names with :: I realized that this was one of the missing pieces in the bash scripts of ours. While we already had packages (in libraries folders) it's been difficult to realise which function is where. With introducing packages - equal to the library file name we are *almost* at a level of a structured language - and it's easier to find the functions if you are looking for them. Way easier in fact. Part of #10576 (cherry picked from commit cc551ba793344800d2d396c13d7fd0c8eed97352) (cherry picked from commit 2bba276f0f06a5981bdd7e4f0e7e5ca2fe84f063)
2020-09-02 22:58:37 +03:00
$(breeze::print_star_line)
Help for breeze commands contain relevant flags. (#8261) * Help for breeze commands contain relevant flags. * fixup! Help for breeze commands contain relevant flags. * fixup! fixup! Help for breeze commands contain relevant flags.
2020-04-13 09:51:33 +03:00
Credentials
Add packages to function names in bash (#10670) (#10696) Inspired by the Google Shell Guide where they mentioned separating package names with :: I realized that this was one of the missing pieces in the bash scripts of ours. While we already had packages (in libraries folders) it's been difficult to realise which function is where. With introducing packages - equal to the library file name we are *almost* at a level of a structured language - and it's easier to find the functions if you are looking for them. Way easier in fact. Part of #10576 (cherry picked from commit cc551ba793344800d2d396c13d7fd0c8eed97352) (cherry picked from commit 2bba276f0f06a5981bdd7e4f0e7e5ca2fe84f063)
2020-09-02 22:58:37 +03:00
$(breeze::flag_credentials)
Help for breeze commands contain relevant flags. (#8261) * Help for breeze commands contain relevant flags. * fixup! Help for breeze commands contain relevant flags. * fixup! fixup! Help for breeze commands contain relevant flags.
2020-04-13 09:51:33 +03:00
Add packages to function names in bash (#10670) (#10696) Inspired by the Google Shell Guide where they mentioned separating package names with :: I realized that this was one of the missing pieces in the bash scripts of ours. While we already had packages (in libraries folders) it's been difficult to realise which function is where. With introducing packages - equal to the library file name we are *almost* at a level of a structured language - and it's easier to find the functions if you are looking for them. Way easier in fact. Part of #10576 (cherry picked from commit cc551ba793344800d2d396c13d7fd0c8eed97352) (cherry picked from commit 2bba276f0f06a5981bdd7e4f0e7e5ca2fe84f063)
2020-09-02 22:58:37 +03:00
$(breeze::print_star_line)
Help for breeze commands contain relevant flags. (#8261) * Help for breeze commands contain relevant flags. * fixup! Help for breeze commands contain relevant flags. * fixup! fixup! Help for breeze commands contain relevant flags.
2020-04-13 09:51:33 +03:00
Flags for building Docker images (both CI and production)
Add packages to function names in bash (#10670) (#10696) Inspired by the Google Shell Guide where they mentioned separating package names with :: I realized that this was one of the missing pieces in the bash scripts of ours. While we already had packages (in libraries folders) it's been difficult to realise which function is where. With introducing packages - equal to the library file name we are *almost* at a level of a structured language - and it's easier to find the functions if you are looking for them. Way easier in fact. Part of #10576 (cherry picked from commit cc551ba793344800d2d396c13d7fd0c8eed97352) (cherry picked from commit 2bba276f0f06a5981bdd7e4f0e7e5ca2fe84f063)
2020-09-02 22:58:37 +03:00
$(breeze::flag_build_docker_images)
Help for breeze commands contain relevant flags. (#8261) * Help for breeze commands contain relevant flags. * fixup! Help for breeze commands contain relevant flags. * fixup! fixup! Help for breeze commands contain relevant flags.
2020-04-13 09:51:33 +03:00
Add packages to function names in bash (#10670) (#10696) Inspired by the Google Shell Guide where they mentioned separating package names with :: I realized that this was one of the missing pieces in the bash scripts of ours. While we already had packages (in libraries folders) it's been difficult to realise which function is where. With introducing packages - equal to the library file name we are *almost* at a level of a structured language - and it's easier to find the functions if you are looking for them. Way easier in fact. Part of #10576 (cherry picked from commit cc551ba793344800d2d396c13d7fd0c8eed97352) (cherry picked from commit 2bba276f0f06a5981bdd7e4f0e7e5ca2fe84f063)
2020-09-02 22:58:37 +03:00
$(breeze::print_star_line)
You can push with Breeze as separate command and to cache (#8976) Breeze had --push-images switch to also push images to repo but it was often needed to build and push images separately. We have now a possibility to push an already built image with separate push-image command instead and also you can choose to push to cache registry in GitHub rather than to DockerHub with --registry-cache switch.
2020-06-02 11:24:47 +03:00
Flags for pulling/pushing Docker images (both CI and production)
Add packages to function names in bash (#10670) (#10696) Inspired by the Google Shell Guide where they mentioned separating package names with :: I realized that this was one of the missing pieces in the bash scripts of ours. While we already had packages (in libraries folders) it's been difficult to realise which function is where. With introducing packages - equal to the library file name we are *almost* at a level of a structured language - and it's easier to find the functions if you are looking for them. Way easier in fact. Part of #10576 (cherry picked from commit cc551ba793344800d2d396c13d7fd0c8eed97352) (cherry picked from commit 2bba276f0f06a5981bdd7e4f0e7e5ca2fe84f063)
2020-09-02 22:58:37 +03:00
$(breeze::flag_pull_push_docker_images)
[AIRFLOW-6489] Separate integrations in tests (#7091) You can now choose which integration you want to start when you run Breeze as well as when CI tests are run. Now by default Breeze and CI runs without integrations, but you can add them via Breeze flags or by environment variables when CI is executed. We have pytest markers now that mark tests that can be run for integration, backend and runtime selected. Also we have now more test jobs - we have separate test run for all non-integration tests (with less memory used by the integrations) and separate jobs that run integration tests only (more memory used for integrations but far less number of tests to run)
2020-01-13 22:47:37 +03:00
Add packages to function names in bash (#10670) (#10696) Inspired by the Google Shell Guide where they mentioned separating package names with :: I realized that this was one of the missing pieces in the bash scripts of ours. While we already had packages (in libraries folders) it's been difficult to realise which function is where. With introducing packages - equal to the library file name we are *almost* at a level of a structured language - and it's easier to find the functions if you are looking for them. Way easier in fact. Part of #10576 (cherry picked from commit cc551ba793344800d2d396c13d7fd0c8eed97352) (cherry picked from commit 2bba276f0f06a5981bdd7e4f0e7e5ca2fe84f063)
2020-09-02 22:58:37 +03:00
$(breeze::print_star_line)
Added automated release notes generation for backport operators (#8807) We have now mechanism to keep release notes updated for the backport operators in an automated way. It really nicely generates all the necessary information: * summary of requirements for each backport package * list of dependencies (including extras to install them) when package depends on other providers packages * table of new hooks/operators/sensors/protocols/secrets * table of moved hooks/operators/sensors/protocols/secrets with information where they were moved from * changelog of all the changes to the provider package (this will be automatically updated with incremental changelog whenever we decide to release separate packages. The system is fully automated - we will be able to produce release notes automatically (per-package) whenever we decide to release new version of the package in the future.
2020-05-15 20:00:15 +03:00
Flags for generation of the backport packages
Add packages to function names in bash (#10670) (#10696) Inspired by the Google Shell Guide where they mentioned separating package names with :: I realized that this was one of the missing pieces in the bash scripts of ours. While we already had packages (in libraries folders) it's been difficult to realise which function is where. With introducing packages - equal to the library file name we are *almost* at a level of a structured language - and it's easier to find the functions if you are looking for them. Way easier in fact. Part of #10576 (cherry picked from commit cc551ba793344800d2d396c13d7fd0c8eed97352) (cherry picked from commit 2bba276f0f06a5981bdd7e4f0e7e5ca2fe84f063)
2020-09-02 22:58:37 +03:00
$(breeze::flag_version_suffix)
Added automated release notes generation for backport operators (#8807) We have now mechanism to keep release notes updated for the backport operators in an automated way. It really nicely generates all the necessary information: * summary of requirements for each backport package * list of dependencies (including extras to install them) when package depends on other providers packages * table of new hooks/operators/sensors/protocols/secrets * table of moved hooks/operators/sensors/protocols/secrets with information where they were moved from * changelog of all the changes to the provider package (this will be automatically updated with incremental changelog whenever we decide to release separate packages. The system is fully automated - we will be able to produce release notes automatically (per-package) whenever we decide to release new version of the package in the future.
2020-05-15 20:00:15 +03:00
Add packages to function names in bash (#10670) (#10696) Inspired by the Google Shell Guide where they mentioned separating package names with :: I realized that this was one of the missing pieces in the bash scripts of ours. While we already had packages (in libraries folders) it's been difficult to realise which function is where. With introducing packages - equal to the library file name we are *almost* at a level of a structured language - and it's easier to find the functions if you are looking for them. Way easier in fact. Part of #10576 (cherry picked from commit cc551ba793344800d2d396c13d7fd0c8eed97352) (cherry picked from commit 2bba276f0f06a5981bdd7e4f0e7e5ca2fe84f063)
2020-09-02 22:58:37 +03:00
$(breeze::print_star_line)
Help for breeze commands contain relevant flags. (#8261) * Help for breeze commands contain relevant flags. * fixup! Help for breeze commands contain relevant flags. * fixup! fixup! Help for breeze commands contain relevant flags.
2020-04-13 09:51:33 +03:00
Increase verbosity of the scripts
Add packages to function names in bash (#10670) (#10696) Inspired by the Google Shell Guide where they mentioned separating package names with :: I realized that this was one of the missing pieces in the bash scripts of ours. While we already had packages (in libraries folders) it's been difficult to realise which function is where. With introducing packages - equal to the library file name we are *almost* at a level of a structured language - and it's easier to find the functions if you are looking for them. Way easier in fact. Part of #10576 (cherry picked from commit cc551ba793344800d2d396c13d7fd0c8eed97352) (cherry picked from commit 2bba276f0f06a5981bdd7e4f0e7e5ca2fe84f063)
2020-09-02 22:58:37 +03:00
$(breeze::flag_verbosity)
Kubernetes Cluster is started on host not in the container (#8265) Tests requiring Kubernetes Cluster are now moved out of the regular CI tests and moved to "kubernetes_tests" folder so that they can be run entirely on host without having the CI image built at all. They use production image to run the tests on KinD cluster and we add tooling to start/stop/deploy the application to the KinD cluster automatically - for both CI testing and local development. This is a pre-requisite to convert the tests to convert the tests to use the official Helm Chart and Docker images or Apache Airflow. It closes #8782
2020-06-03 21:58:38 +03:00
Add packages to function names in bash (#10670) (#10696) Inspired by the Google Shell Guide where they mentioned separating package names with :: I realized that this was one of the missing pieces in the bash scripts of ours. While we already had packages (in libraries folders) it's been difficult to realise which function is where. With introducing packages - equal to the library file name we are *almost* at a level of a structured language - and it's easier to find the functions if you are looking for them. Way easier in fact. Part of #10576 (cherry picked from commit cc551ba793344800d2d396c13d7fd0c8eed97352) (cherry picked from commit 2bba276f0f06a5981bdd7e4f0e7e5ca2fe84f063)
2020-09-02 22:58:37 +03:00
$(breeze::print_star_line)
Kubernetes Cluster is started on host not in the container (#8265) Tests requiring Kubernetes Cluster are now moved out of the regular CI tests and moved to "kubernetes_tests" folder so that they can be run entirely on host without having the CI image built at all. They use production image to run the tests on KinD cluster and we add tooling to start/stop/deploy the application to the KinD cluster automatically - for both CI testing and local development. This is a pre-requisite to convert the tests to convert the tests to use the official Helm Chart and Docker images or Apache Airflow. It closes #8782
2020-06-03 21:58:38 +03:00
Print detailed help message
Add packages to function names in bash (#10670) (#10696) Inspired by the Google Shell Guide where they mentioned separating package names with :: I realized that this was one of the missing pieces in the bash scripts of ours. While we already had packages (in libraries folders) it's been difficult to realise which function is where. With introducing packages - equal to the library file name we are *almost* at a level of a structured language - and it's easier to find the functions if you are looking for them. Way easier in fact. Part of #10576 (cherry picked from commit cc551ba793344800d2d396c13d7fd0c8eed97352) (cherry picked from commit 2bba276f0f06a5981bdd7e4f0e7e5ca2fe84f063)
2020-09-02 22:58:37 +03:00
$(breeze::flag_help)
[AIRFLOW-6838] Introduce real subcommands for Breeze (#7515) This change introduces sub-commands in breeze tool. It is much needed as we have many commands now and it was difficult to separate commands from flags. Also --help output was very long and unreadable. With this change help it is much easier to discover what breeze can do for you as well as navigate with it. Co-authored-by: Jarek Potiuk <jarek@potiuk.com> Co-authored-by: Kamil Breguła <mik-laj@users.noreply.github.com>
2020-02-25 00:31:50 +03:00
"
[AIRFLOW-3611] Simplified development environment (#4932)
2019-08-27 21:39:36 +03:00
}
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
#######################################################################################################
#
# Prints header line filling screen indented_screen_width - only when VERBOSE is set
#
# Outputs:
# Prints header line.
#######################################################################################################
Add packages to function names in bash (#10670) (#10696) Inspired by the Google Shell Guide where they mentioned separating package names with :: I realized that this was one of the missing pieces in the bash scripts of ours. While we already had packages (in libraries folders) it's been difficult to realise which function is where. With introducing packages - equal to the library file name we are *almost* at a level of a structured language - and it's easier to find the functions if you are looking for them. Way easier in fact. Part of #10576 (cherry picked from commit cc551ba793344800d2d396c13d7fd0c8eed97352) (cherry picked from commit 2bba276f0f06a5981bdd7e4f0e7e5ca2fe84f063)
2020-09-02 22:58:37 +03:00
function breeze::print_header_line() {
[AIRFLOW-6838] Introduce real subcommands for Breeze (#7515) This change introduces sub-commands in breeze tool. It is much needed as we have many commands now and it was difficult to separate commands from flags. Also --help output was very long and unreadable. With this change help it is much easier to discover what breeze can do for you as well as navigate with it. Co-authored-by: Jarek Potiuk <jarek@potiuk.com> Co-authored-by: Kamil Breguła <mik-laj@users.noreply.github.com>
2020-02-25 00:31:50 +03:00
if [ ${VERBOSE:="false"} == "true" ]; then
echo
printf '=%.0s' $(seq "${SCREEN_WIDTH}")
echo
fi
[AIRFLOW-6820] split breeze into functions (#7433)
2020-02-19 03:58:53 +03:00
}
[AIRFLOW-3611] Simplified development environment (#4932)
2019-08-27 21:39:36 +03:00
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
#######################################################################################################
#
# Prints separation line filling screen indented_screen_width
#
# Outputs:
# Prints line.
#######################################################################################################
Add packages to function names in bash (#10670) (#10696) Inspired by the Google Shell Guide where they mentioned separating package names with :: I realized that this was one of the missing pieces in the bash scripts of ours. While we already had packages (in libraries folders) it's been difficult to realise which function is where. With introducing packages - equal to the library file name we are *almost* at a level of a structured language - and it's easier to find the functions if you are looking for them. Way easier in fact. Part of #10576 (cherry picked from commit cc551ba793344800d2d396c13d7fd0c8eed97352) (cherry picked from commit 2bba276f0f06a5981bdd7e4f0e7e5ca2fe84f063)
2020-09-02 22:58:37 +03:00
function breeze::print_line() {
[AIRFLOW-6838] Introduce real subcommands for Breeze (#7515) This change introduces sub-commands in breeze tool. It is much needed as we have many commands now and it was difficult to separate commands from flags. Also --help output was very long and unreadable. With this change help it is much easier to discover what breeze can do for you as well as navigate with it. Co-authored-by: Jarek Potiuk <jarek@potiuk.com> Co-authored-by: Kamil Breguła <mik-laj@users.noreply.github.com>
2020-02-25 00:31:50 +03:00
printf '#%.0s' $(seq "${SCREEN_WIDTH}")
}
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
# Prints star line filling screen indented_screen_width
Add packages to function names in bash (#10670) (#10696) Inspired by the Google Shell Guide where they mentioned separating package names with :: I realized that this was one of the missing pieces in the bash scripts of ours. While we already had packages (in libraries folders) it's been difficult to realise which function is where. With introducing packages - equal to the library file name we are *almost* at a level of a structured language - and it's easier to find the functions if you are looking for them. Way easier in fact. Part of #10576 (cherry picked from commit cc551ba793344800d2d396c13d7fd0c8eed97352) (cherry picked from commit 2bba276f0f06a5981bdd7e4f0e7e5ca2fe84f063)
2020-09-02 22:58:37 +03:00
function breeze::print_star_line() {
[AIRFLOW-6838] Introduce real subcommands for Breeze (#7515) This change introduces sub-commands in breeze tool. It is much needed as we have many commands now and it was difficult to separate commands from flags. Also --help output was very long and unreadable. With this change help it is much easier to discover what breeze can do for you as well as navigate with it. Co-authored-by: Jarek Potiuk <jarek@potiuk.com> Co-authored-by: Kamil Breguła <mik-laj@users.noreply.github.com>
2020-02-25 00:31:50 +03:00
printf '*%.0s' $(seq "${SCREEN_WIDTH}")
[AIRFLOW-6820] split breeze into functions (#7433)
2020-02-19 03:58:53 +03:00
}
[AIRFLOW-3611] Simplified development environment (#4932)
2019-08-27 21:39:36 +03:00
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
#######################################################################################################
#
# Reads saved environment variables. Some of the variables are stored across session so that once
# you use them you do not have to use it next time. That makes those flags persistent.
#
# An example of it is `--backend` or '--kubernetes-mode' flags.
#
# Note that PYTHON_MAJOR_MINOR_VERSION is not read here - it is read at the
# `setup_default_breeze_variables` method because it is needed
# to determine the right images to use and set several variables that depend on the Python version
#
# Uses:
Make breeeze-complete Google Shell Guide compatible (#10708) Also added unit tests for breeze-complete Part of #10576
2020-09-14 11:21:09 +03:00
# _breeze_default_* corresponding defaults for variables it reads
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
#
# Modified and used global constants:
#
# BACKEND
# KUBERNETES_MODE
# KUBERNETES_VERSION
# KIND_VERSION
# HELM_VERSION
# POSTGRES_VERSION
# MYSQL_VERSION
# DOCKERHUB_USER
# DOCKERHUB_REPO
#
#######################################################################################################
Add packages to function names in bash (#10670) (#10696) Inspired by the Google Shell Guide where they mentioned separating package names with :: I realized that this was one of the missing pieces in the bash scripts of ours. While we already had packages (in libraries folders) it's been difficult to realise which function is where. With introducing packages - equal to the library file name we are *almost* at a level of a structured language - and it's easier to find the functions if you are looking for them. Way easier in fact. Part of #10576 (cherry picked from commit cc551ba793344800d2d396c13d7fd0c8eed97352) (cherry picked from commit 2bba276f0f06a5981bdd7e4f0e7e5ca2fe84f063)
2020-09-02 22:58:37 +03:00
function breeze::read_saved_environment_variables() {
BACKEND="${BACKEND:=$(parameters::read_from_file BACKEND)}"
Make breeeze-complete Google Shell Guide compatible (#10708) Also added unit tests for breeze-complete Part of #10576
2020-09-14 11:21:09 +03:00
BACKEND=${BACKEND:-${_breeze_default_backend}}
[AIRFLOW-5369] Adds interactivity to pre-commits (#5976) This commit adds full interactivity to pre-commits. Whenever you run pre-commit and it detects that the image should be rebuild, an interactive question will pop up instead of failing the build and asking to rebuild with REBUILD=yes This is much nicer from the user perspective. You can choose whether to: 1) Rebuild the image (which will take some time) 2) Not rebuild the image (this will use the old image with hope it's OK) 3) Quit. Answer to that question is carried across all images needed to rebuild. There is the special "build" pre-commit hook that takes care about that. Note that this interactive question cannot be asked if you run only single pre-commit hook with Dockerfile because it can run multiple processes and you can start building in parallel. This is not desired so instead we fail such builds.
2019-09-18 14:43:30 +03:00
Add packages to function names in bash (#10670) (#10696) Inspired by the Google Shell Guide where they mentioned separating package names with :: I realized that this was one of the missing pieces in the bash scripts of ours. While we already had packages (in libraries folders) it's been difficult to realise which function is where. With introducing packages - equal to the library file name we are *almost* at a level of a structured language - and it's easier to find the functions if you are looking for them. Way easier in fact. Part of #10576 (cherry picked from commit cc551ba793344800d2d396c13d7fd0c8eed97352) (cherry picked from commit 2bba276f0f06a5981bdd7e4f0e7e5ca2fe84f063)
2020-09-02 22:58:37 +03:00
KUBERNETES_MODE="${KUBERNETES_MODE:=$(parameters::read_from_file KUBERNETES_MODE)}"
Make breeeze-complete Google Shell Guide compatible (#10708) Also added unit tests for breeze-complete Part of #10576
2020-09-14 11:21:09 +03:00
KUBERNETES_MODE=${KUBERNETES_MODE:=${_breeze_default_kubernetes_mode}}
[AIRFLOW-5704] Improve Kind Kubernetes scripts for local testing (#6516) * Fixed problem that Kubernetes tests were testing latest master rather than what came from the local sources. * Kind (Kubernetes in Dcocker) is run in the same Docker as Breeze env * Moved Kubernetes scripts to 'in_container' dir where they belong now * Kubernetes cluster is reused until it is stopped * Kubernetes image is build from image in docker already + mounted sources * Kubectl version name is corrected in the Dockerfile * KUBERNETES_VERSION can now be used to select Kubernetes version * Running kubernetes scripts is now easy in Breeze * We can start/recreate/stop cluster using --<ACTION>-kind-cluster * Instructions on how to run Kubernetes tests are updated * The old "bare" environment is replaced by --no-deps switch
2020-01-11 18:25:19 +03:00
Add packages to function names in bash (#10670) (#10696) Inspired by the Google Shell Guide where they mentioned separating package names with :: I realized that this was one of the missing pieces in the bash scripts of ours. While we already had packages (in libraries folders) it's been difficult to realise which function is where. With introducing packages - equal to the library file name we are *almost* at a level of a structured language - and it's easier to find the functions if you are looking for them. Way easier in fact. Part of #10576 (cherry picked from commit cc551ba793344800d2d396c13d7fd0c8eed97352) (cherry picked from commit 2bba276f0f06a5981bdd7e4f0e7e5ca2fe84f063)
2020-09-02 22:58:37 +03:00
KUBERNETES_VERSION="${KUBERNETES_VERSION:=$(parameters::read_from_file KUBERNETES_VERSION)}"
Make breeeze-complete Google Shell Guide compatible (#10708) Also added unit tests for breeze-complete Part of #10576
2020-09-14 11:21:09 +03:00
KUBERNETES_VERSION=${KUBERNETES_VERSION:=${_breeze_default_kubernetes_version}}
[AIRFLOW-5704] Improve Kind Kubernetes scripts for local testing (#6516) * Fixed problem that Kubernetes tests were testing latest master rather than what came from the local sources. * Kind (Kubernetes in Dcocker) is run in the same Docker as Breeze env * Moved Kubernetes scripts to 'in_container' dir where they belong now * Kubernetes cluster is reused until it is stopped * Kubernetes image is build from image in docker already + mounted sources * Kubectl version name is corrected in the Dockerfile * KUBERNETES_VERSION can now be used to select Kubernetes version * Running kubernetes scripts is now easy in Breeze * We can start/recreate/stop cluster using --<ACTION>-kind-cluster * Instructions on how to run Kubernetes tests are updated * The old "bare" environment is replaced by --no-deps switch
2020-01-11 18:25:19 +03:00
Add packages to function names in bash (#10670) (#10696) Inspired by the Google Shell Guide where they mentioned separating package names with :: I realized that this was one of the missing pieces in the bash scripts of ours. While we already had packages (in libraries folders) it's been difficult to realise which function is where. With introducing packages - equal to the library file name we are *almost* at a level of a structured language - and it's easier to find the functions if you are looking for them. Way easier in fact. Part of #10576 (cherry picked from commit cc551ba793344800d2d396c13d7fd0c8eed97352) (cherry picked from commit 2bba276f0f06a5981bdd7e4f0e7e5ca2fe84f063)
2020-09-02 22:58:37 +03:00
KIND_VERSION="${KIND_VERSION:=$(parameters::read_from_file KIND_VERSION)}"
Make breeeze-complete Google Shell Guide compatible (#10708) Also added unit tests for breeze-complete Part of #10576
2020-09-14 11:21:09 +03:00
KIND_VERSION=${KIND_VERSION:=${_breeze_default_kind_version}}
Switches to Helm Chart for Kubernetes tests (#9468) The Kubernetes tests are now run using Helm chart rather than the custom templates we used to have. The Helm Chart uses locally build production image so the tests are testing not only Airflow but also Helm Chart and a Production image - all at the same time. Later on we will add more tests covering more functionalities of both Helm Chart and Production Image. This is the first step to get all of those bundle together and become testable. This change introduces also 'shell' sub-command for Breeze's kind-cluster command and EMBEDDED_DAGS build args for production image - both of them useful to run the Kubernetes tests more easily - without building two images and with an easy-to-iterate-over-tests shell command - which works without any other development environment. Co-authored-by: Jarek Potiuk <jarek@potiuk.com> Co-authored-by: Daniel Imberman <daniel@astronomer.io>
2020-07-01 15:50:30 +03:00
Add packages to function names in bash (#10670) (#10696) Inspired by the Google Shell Guide where they mentioned separating package names with :: I realized that this was one of the missing pieces in the bash scripts of ours. While we already had packages (in libraries folders) it's been difficult to realise which function is where. With introducing packages - equal to the library file name we are *almost* at a level of a structured language - and it's easier to find the functions if you are looking for them. Way easier in fact. Part of #10576 (cherry picked from commit cc551ba793344800d2d396c13d7fd0c8eed97352) (cherry picked from commit 2bba276f0f06a5981bdd7e4f0e7e5ca2fe84f063)
2020-09-02 22:58:37 +03:00
HELM_VERSION="${HELM_VERSION:=$(parameters::read_from_file HELM_VERSION)}"
Make breeeze-complete Google Shell Guide compatible (#10708) Also added unit tests for breeze-complete Part of #10576
2020-09-14 11:21:09 +03:00
HELM_VERSION=${HELM_VERSION:=${_breeze_default_helm_version}}
Switches to Helm Chart for Kubernetes tests (#9468) The Kubernetes tests are now run using Helm chart rather than the custom templates we used to have. The Helm Chart uses locally build production image so the tests are testing not only Airflow but also Helm Chart and a Production image - all at the same time. Later on we will add more tests covering more functionalities of both Helm Chart and Production Image. This is the first step to get all of those bundle together and become testable. This change introduces also 'shell' sub-command for Breeze's kind-cluster command and EMBEDDED_DAGS build args for production image - both of them useful to run the Kubernetes tests more easily - without building two images and with an easy-to-iterate-over-tests shell command - which works without any other development environment. Co-authored-by: Jarek Potiuk <jarek@potiuk.com> Co-authored-by: Daniel Imberman <daniel@astronomer.io>
2020-07-01 15:50:30 +03:00
Add packages to function names in bash (#10670) (#10696) Inspired by the Google Shell Guide where they mentioned separating package names with :: I realized that this was one of the missing pieces in the bash scripts of ours. While we already had packages (in libraries folders) it's been difficult to realise which function is where. With introducing packages - equal to the library file name we are *almost* at a level of a structured language - and it's easier to find the functions if you are looking for them. Way easier in fact. Part of #10576 (cherry picked from commit cc551ba793344800d2d396c13d7fd0c8eed97352) (cherry picked from commit 2bba276f0f06a5981bdd7e4f0e7e5ca2fe84f063)
2020-09-02 22:58:37 +03:00
POSTGRES_VERSION="${POSTGRES_VERSION:=$(parameters::read_from_file POSTGRES_VERSION)}"
Make breeeze-complete Google Shell Guide compatible (#10708) Also added unit tests for breeze-complete Part of #10576
2020-09-14 11:21:09 +03:00
POSTGRES_VERSION=${POSTGRES_VERSION:=${_breeze_default_postgres_version}}
[AIRFLOW-7058] Add support for different DB versions (#7717)
2020-03-15 01:57:47 +03:00
Add packages to function names in bash (#10670) (#10696) Inspired by the Google Shell Guide where they mentioned separating package names with :: I realized that this was one of the missing pieces in the bash scripts of ours. While we already had packages (in libraries folders) it's been difficult to realise which function is where. With introducing packages - equal to the library file name we are *almost* at a level of a structured language - and it's easier to find the functions if you are looking for them. Way easier in fact. Part of #10576 (cherry picked from commit cc551ba793344800d2d396c13d7fd0c8eed97352) (cherry picked from commit 2bba276f0f06a5981bdd7e4f0e7e5ca2fe84f063)
2020-09-02 22:58:37 +03:00
MYSQL_VERSION="${MYSQL_VERSION:=$(parameters::read_from_file MYSQL_VERSION)}"
Make breeeze-complete Google Shell Guide compatible (#10708) Also added unit tests for breeze-complete Part of #10576
2020-09-14 11:21:09 +03:00
MYSQL_VERSION=${MYSQL_VERSION:=${_breeze_default_mysql_version}}
[AIRFLOW-7058] Add support for different DB versions (#7717)
2020-03-15 01:57:47 +03:00
[AIRFLOW-6820] split breeze into functions (#7433)
2020-02-19 03:58:53 +03:00
# Here you read DockerHub user/account that you use
# You can populate your own images in DockerHub this way and work with the,
Add comments to breeze scripts (#8797)
2020-05-10 20:55:40 +03:00
# You can override it with "--dockerhub-user" option and it will be stored in .build directory
Add packages to function names in bash (#10670) (#10696) Inspired by the Google Shell Guide where they mentioned separating package names with :: I realized that this was one of the missing pieces in the bash scripts of ours. While we already had packages (in libraries folders) it's been difficult to realise which function is where. With introducing packages - equal to the library file name we are *almost* at a level of a structured language - and it's easier to find the functions if you are looking for them. Way easier in fact. Part of #10576 (cherry picked from commit cc551ba793344800d2d396c13d7fd0c8eed97352) (cherry picked from commit 2bba276f0f06a5981bdd7e4f0e7e5ca2fe84f063)
2020-09-02 22:58:37 +03:00
DOCKERHUB_USER="${DOCKERHUB_USER:=$(parameters::read_from_file DOCKERHUB_USER)}"
Make breeeze-complete Google Shell Guide compatible (#10708) Also added unit tests for breeze-complete Part of #10576
2020-09-14 11:21:09 +03:00
DOCKERHUB_USER="${DOCKERHUB_USER:=${_breeze_default_dockerhub_user}}"
[AIRFLOW-3611] Simplified development environment (#4932)
2019-08-27 21:39:36 +03:00
[AIRFLOW-6820] split breeze into functions (#7433)
2020-02-19 03:58:53 +03:00
# Here you read DockerHub repo that you use
# You can populate your own images in DockerHub this way and work with them
Add comments to breeze scripts (#8797)
2020-05-10 20:55:40 +03:00
# You can override it with "--dockerhub-repo" option and it will be stored in .build directory
Add packages to function names in bash (#10670) (#10696) Inspired by the Google Shell Guide where they mentioned separating package names with :: I realized that this was one of the missing pieces in the bash scripts of ours. While we already had packages (in libraries folders) it's been difficult to realise which function is where. With introducing packages - equal to the library file name we are *almost* at a level of a structured language - and it's easier to find the functions if you are looking for them. Way easier in fact. Part of #10576 (cherry picked from commit cc551ba793344800d2d396c13d7fd0c8eed97352) (cherry picked from commit 2bba276f0f06a5981bdd7e4f0e7e5ca2fe84f063)
2020-09-02 22:58:37 +03:00
DOCKERHUB_REPO="${DOCKERHUB_REPO:=$(parameters::read_from_file DOCKERHUB_REPO)}"
Make breeeze-complete Google Shell Guide compatible (#10708) Also added unit tests for breeze-complete Part of #10576
2020-09-14 11:21:09 +03:00
DOCKERHUB_REPO="${DOCKERHUB_REPO:=${_breeze_default_dockerhub_repo}}"
[AIRFLOW-6820] split breeze into functions (#7433)
2020-02-19 03:58:53 +03:00
}
[AIRFLOW-3611] Simplified development environment (#4932)
2019-08-27 21:39:36 +03:00
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
#######################################################################################################
#
Add comments to breeze scripts (#8797)
2020-05-10 20:55:40 +03:00
# Checks if variables are correctly set and if they are - saves them so that they can be used across
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
# sessions. In case we are installing Airflow 1.10, the constants are set to match 1.10 line.
#
# In case the variables are matching expected values they are saved in ".build/VARIABLE_NAME" for
# later reuse. If not, error is printed and the saved file is cleaned, so that next time
# default value can be used.
#
# Used Global constants:
# INSTALL_AIRFLOW_REFERENCE
# INSTALL_AIRFLOW_VERSION
# BACKEND
# KUBERNETES_MODE
# KUBERNETES_VERSION
# KIND_VERSION
# HELM_VERSION
# POSTGRES_VERSION
# MYSQL_VERSION
# DOCKERHUB_USER
# DOCKERHUB_REPO
#
# Updated Global constants:
# BRANCH_NAME
#
# Output: saved variable files in .build,
#######################################################################################################
Add packages to function names in bash (#10670) (#10696) Inspired by the Google Shell Guide where they mentioned separating package names with :: I realized that this was one of the missing pieces in the bash scripts of ours. While we already had packages (in libraries folders) it's been difficult to realise which function is where. With introducing packages - equal to the library file name we are *almost* at a level of a structured language - and it's easier to find the functions if you are looking for them. Way easier in fact. Part of #10576 (cherry picked from commit cc551ba793344800d2d396c13d7fd0c8eed97352) (cherry picked from commit 2bba276f0f06a5981bdd7e4f0e7e5ca2fe84f063)
2020-09-02 22:58:37 +03:00
function breeze::check_and_save_all_params() {
parameters::check_and_save_allowed_param "PYTHON_MAJOR_MINOR_VERSION" "Python version" "--python"
Improve production image iteration speed (#9162) For a long time the way how entrypoint worked in ci scripts was wrong. The way it worked was convoluted and short of black magic. This did not allow to pass multiple test targets and required separate execute command scripts in Breeze. This is all now straightened out and both production and CI image are always using the right entrypoint by default and we can simply pass parameters to the image as usual without escaping strings. This also allowed to remove some breeze commands and change names of several flags in Breeze to make them more meaningful. Both CI and PROD image have now embedded scripts for log cleaning. History of image releases is added for 1.10.10-* alpha quality images.
2020-06-16 13:36:46 +03:00
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
if [[ -n "${INSTALL_AIRFLOW_REFERENCE=}" ]]; then
if [[ ${INSTALL_AIRFLOW_REFERENCE=} == *1_10* ]]; then
Improve production image iteration speed (#9162) For a long time the way how entrypoint worked in ci scripts was wrong. The way it worked was convoluted and short of black magic. This did not allow to pass multiple test targets and required separate execute command scripts in Breeze. This is all now straightened out and both production and CI image are always using the right entrypoint by default and we can simply pass parameters to the image as usual without escaping strings. This also allowed to remove some breeze commands and change names of several flags in Breeze to make them more meaningful. Both CI and PROD image have now embedded scripts for log cleaning. History of image releases is added for 1.10.10-* alpha quality images.
2020-06-16 13:36:46 +03:00
export BRANCH_NAME="v1-10-test"
fi
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
elif [[ -n "${INSTALL_AIRFLOW_VERSION=}" ]]; then
if [[ ${INSTALL_AIRFLOW_VERSION=} == *1.10* ]]; then
Improve production image iteration speed (#9162) For a long time the way how entrypoint worked in ci scripts was wrong. The way it worked was convoluted and short of black magic. This did not allow to pass multiple test targets and required separate execute command scripts in Breeze. This is all now straightened out and both production and CI image are always using the right entrypoint by default and we can simply pass parameters to the image as usual without escaping strings. This also allowed to remove some breeze commands and change names of several flags in Breeze to make them more meaningful. Both CI and PROD image have now embedded scripts for log cleaning. History of image releases is added for 1.10.10-* alpha quality images.
2020-06-16 13:36:46 +03:00
export BRANCH_NAME="v1-10-test"
fi
fi
if [[ ${PYTHON_MAJOR_MINOR_VERSION} == "2.7" || ${PYTHON_MAJOR_MINOR_VERSION} == "3.5" ]]; then
if [[ ${BRANCH_NAME} == "master" ]]; then
echo >&2
echo >&2 "The ${PYTHON_MAJOR_MINOR_VERSION} can only be used when installing Airflow 1.10.*"
echo >&2
echo >&2 "You can use it only when you specify 1.10 Airflow via --install-airflow-version"
echo >&2 "or --install-airflow-reference and they point to 1.10 version of Airflow"
echo >&2
exit 1
fi
fi
Add packages to function names in bash (#10670) (#10696) Inspired by the Google Shell Guide where they mentioned separating package names with :: I realized that this was one of the missing pieces in the bash scripts of ours. While we already had packages (in libraries folders) it's been difficult to realise which function is where. With introducing packages - equal to the library file name we are *almost* at a level of a structured language - and it's easier to find the functions if you are looking for them. Way easier in fact. Part of #10576 (cherry picked from commit cc551ba793344800d2d396c13d7fd0c8eed97352) (cherry picked from commit 2bba276f0f06a5981bdd7e4f0e7e5ca2fe84f063)
2020-09-02 22:58:37 +03:00
parameters::check_and_save_allowed_param "BACKEND" "backend" "--backend"
parameters::check_and_save_allowed_param "KUBERNETES_MODE" "Kubernetes mode" "--kubernetes-mode"
parameters::check_and_save_allowed_param "KUBERNETES_VERSION" "Kubernetes version" "--kubernetes-version"
parameters::check_and_save_allowed_param "KIND_VERSION" "KinD version" "--kind-version"
parameters::check_and_save_allowed_param "HELM_VERSION" "Helm version" "--helm-version"
parameters::check_and_save_allowed_param "POSTGRES_VERSION" "Postgres version" "--postgres-version"
parameters::check_and_save_allowed_param "MYSQL_VERSION" "Mysql version" "--mysql-version"
[AIRFLOW-3611] Simplified development environment (#4932)
2019-08-27 21:39:36 +03:00
Add comments to breeze scripts (#8797)
2020-05-10 20:55:40 +03:00
# Can't verify those - they can be anything, so let's just save them
Add packages to function names in bash (#10670) (#10696) Inspired by the Google Shell Guide where they mentioned separating package names with :: I realized that this was one of the missing pieces in the bash scripts of ours. While we already had packages (in libraries folders) it's been difficult to realise which function is where. With introducing packages - equal to the library file name we are *almost* at a level of a structured language - and it's easier to find the functions if you are looking for them. Way easier in fact. Part of #10576 (cherry picked from commit cc551ba793344800d2d396c13d7fd0c8eed97352) (cherry picked from commit 2bba276f0f06a5981bdd7e4f0e7e5ca2fe84f063)
2020-09-02 22:58:37 +03:00
parameters::save_to_file DOCKERHUB_USER
parameters::save_to_file DOCKERHUB_REPO
[AIRFLOW-6820] split breeze into functions (#7433)
2020-02-19 03:58:53 +03:00
}
[AIRFLOW-5704] Improve Kind Kubernetes scripts for local testing (#6496) * [AIRFLOW-5704] Improve Kind Kubernetes scripts for local testing * Fixed problem that Kubernetes tests were testing latest master rather than what came from the local sources. * Moved Kubernetes scripts to 'in_container' dir where they belong now * Kubernetes tests are now better suited for running locally * Kubernetes cluster is not deleted until environment is stopped * Kubernetes image is built outside of the container and passed as .tar * Kubectl version name is corrected in the Dockerfile * Kubernetes Version can be used to select Kubernetes versio * Running kubernetes scripts is now easy in Breeze * Instructions on how to run Kubernetes tests are updated * Better flags in Breeze are used to run Kubernetes environment/tests * The old "bare" environment is replaced by --no-deps switch
2019-11-06 19:25:11 +03:00
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
#######################################################################################################
#
Add comments to breeze scripts (#8797)
2020-05-10 20:55:40 +03:00
# Prints cheatsheet if it is not suppressed
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
#
# Used global constants:
#
# AIRFLOW_SOURCES
# WEBSERVER_HOST_PORT
# POSTGRES_HOST_PORT
# MYSQL_HOST_PORT
#
#######################################################################################################
Add packages to function names in bash (#10670) (#10696) Inspired by the Google Shell Guide where they mentioned separating package names with :: I realized that this was one of the missing pieces in the bash scripts of ours. While we already had packages (in libraries folders) it's been difficult to realise which function is where. With introducing packages - equal to the library file name we are *almost* at a level of a structured language - and it's easier to find the functions if you are looking for them. Way easier in fact. Part of #10576 (cherry picked from commit cc551ba793344800d2d396c13d7fd0c8eed97352) (cherry picked from commit 2bba276f0f06a5981bdd7e4f0e7e5ca2fe84f063)
2020-09-02 22:58:37 +03:00
function breeze::print_cheatsheet() {
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
if [[ ! -f ${SUPPRESS_CHEATSHEET_FILE} && ${command_to_run} == "enter_breeze" ]]; then
New Breeze command start-airflow, it replaces the previous flag (#11157)
2020-09-27 19:31:50 +03:00
[AIRFLOW-3611] Simplified development environment (#4932)
2019-08-27 21:39:36 +03:00
echo
Add packages to function names in bash (#10670) (#10696) Inspired by the Google Shell Guide where they mentioned separating package names with :: I realized that this was one of the missing pieces in the bash scripts of ours. While we already had packages (in libraries folders) it's been difficult to realise which function is where. With introducing packages - equal to the library file name we are *almost* at a level of a structured language - and it's easier to find the functions if you are looking for them. Way easier in fact. Part of #10576 (cherry picked from commit cc551ba793344800d2d396c13d7fd0c8eed97352) (cherry picked from commit 2bba276f0f06a5981bdd7e4f0e7e5ca2fe84f063)
2020-09-02 22:58:37 +03:00
breeze::print_line
[AIRFLOW-3611] Simplified development environment (#4932)
2019-08-27 21:39:36 +03:00
echo
[AIRFLOW-6820] split breeze into functions (#7433)
2020-02-19 03:58:53 +03:00
echo " Airflow Breeze CHEATSHEET"
[AIRFLOW-3611] Simplified development environment (#4932)
2019-08-27 21:39:36 +03:00
echo
[AIRFLOW-6820] split breeze into functions (#7433)
2020-02-19 03:58:53 +03:00
set +e
if ! command -v breeze; then
Add packages to function names in bash (#10670) (#10696) Inspired by the Google Shell Guide where they mentioned separating package names with :: I realized that this was one of the missing pieces in the bash scripts of ours. While we already had packages (in libraries folders) it's been difficult to realise which function is where. With introducing packages - equal to the library file name we are *almost* at a level of a structured language - and it's easier to find the functions if you are looking for them. Way easier in fact. Part of #10576 (cherry picked from commit cc551ba793344800d2d396c13d7fd0c8eed97352) (cherry picked from commit 2bba276f0f06a5981bdd7e4f0e7e5ca2fe84f063)
2020-09-02 22:58:37 +03:00
breeze::print_line
[AIRFLOW-6820] split breeze into functions (#7433)
2020-02-19 03:58:53 +03:00
echo
echo " Adding breeze to your path:"
Help for breeze commands contain relevant flags. (#8261) * Help for breeze commands contain relevant flags. * fixup! Help for breeze commands contain relevant flags. * fixup! fixup! Help for breeze commands contain relevant flags.
2020-04-13 09:51:33 +03:00
echo " When you exit the environment, you can add sources of Airflow to the path - you can"
[AIRFLOW-6820] split breeze into functions (#7433)
2020-02-19 03:58:53 +03:00
echo " run breeze or the scripts above from any directory by calling 'breeze' commands directly"
echo
Don't reuse MY_DIR in breeze to mean different folder from ci/_utils.sh (#9098) scripts/ci/*.sh uses MY_DIR to mean scripts/ci, but in `breeze` MY_DIR is the same as AIRFLOW_SOURCES. When jumping back-and-forth between ci/_utils.sh, breeze, and ci/ci_*.sh it can be confusing to keep track of what is what. This changes `breeze` to use `AIRFLOW_SOURCES` to referrer to the top level folder instead -- that means I don't have to keep as much context in my head
2020-06-02 09:39:42 +03:00
echo " export PATH=\${PATH}:\"${AIRFLOW_SOURCES}\""
[AIRFLOW-6820] split breeze into functions (#7433)
2020-02-19 03:58:53 +03:00
echo
fi
set -e
Add packages to function names in bash (#10670) (#10696) Inspired by the Google Shell Guide where they mentioned separating package names with :: I realized that this was one of the missing pieces in the bash scripts of ours. While we already had packages (in libraries folders) it's been difficult to realise which function is where. With introducing packages - equal to the library file name we are *almost* at a level of a structured language - and it's easier to find the functions if you are looking for them. Way easier in fact. Part of #10576 (cherry picked from commit cc551ba793344800d2d396c13d7fd0c8eed97352) (cherry picked from commit 2bba276f0f06a5981bdd7e4f0e7e5ca2fe84f063)
2020-09-02 22:58:37 +03:00
breeze::print_line
[AIRFLOW-6820] split breeze into functions (#7433)
2020-02-19 03:58:53 +03:00
[AIRFLOW-3611] Simplified development environment (#4932)
2019-08-27 21:39:36 +03:00
echo
[AIRFLOW-6820] split breeze into functions (#7433)
2020-02-19 03:58:53 +03:00
echo " Port forwarding:"
[AIRFLOW-3611] Simplified development environment (#4932)
2019-08-27 21:39:36 +03:00
echo
[AIRFLOW-6820] split breeze into functions (#7433)
2020-02-19 03:58:53 +03:00
echo " Ports are forwarded to the running docker containers for webserver and database"
Help for breeze commands contain relevant flags. (#8261) * Help for breeze commands contain relevant flags. * fixup! Help for breeze commands contain relevant flags. * fixup! fixup! Help for breeze commands contain relevant flags.
2020-04-13 09:51:33 +03:00
echo " * ${WEBSERVER_HOST_PORT} -> forwarded to Airflow webserver -> airflow:8080"
echo " * ${POSTGRES_HOST_PORT} -> forwarded to Postgres database -> postgres:5432"
echo " * ${MYSQL_HOST_PORT} -> forwarded to MySQL database -> mysql:3306"
[AIRFLOW-3611] Simplified development environment (#4932)
2019-08-27 21:39:36 +03:00
echo
[AIRFLOW-6820] split breeze into functions (#7433)
2020-02-19 03:58:53 +03:00
echo " Here are links to those services that you can use on host:"
echo " * Webserver: http://127.0.0.1:28080"
echo " * Postgres: jdbc:postgresql://127.0.0.1:25433/airflow?user=postgres&password=airflow"
echo " * Mysql: jdbc:mysql://localhost:23306/airflow?user=root"
[AIRFLOW-3611] Simplified development environment (#4932)
2019-08-27 21:39:36 +03:00
echo
[AIRFLOW-6820] split breeze into functions (#7433)
2020-02-19 03:58:53 +03:00
else
echo
fi
}
[AIRFLOW-3611] Simplified development environment (#4932)
2019-08-27 21:39:36 +03:00
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
#######################################################################################################
#
Add comments to breeze scripts (#8797)
2020-05-10 20:55:40 +03:00
# Prints setup instruction in case we find that autocomplete is not set
# also prints how to toggle asciiart/cheatsheet
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
#
# Used global constants:
# CMDNAME
#######################################################################################################
Add packages to function names in bash (#10670) (#10696) Inspired by the Google Shell Guide where they mentioned separating package names with :: I realized that this was one of the missing pieces in the bash scripts of ours. While we already had packages (in libraries folders) it's been difficult to realise which function is where. With introducing packages - equal to the library file name we are *almost* at a level of a structured language - and it's easier to find the functions if you are looking for them. Way easier in fact. Part of #10576 (cherry picked from commit cc551ba793344800d2d396c13d7fd0c8eed97352) (cherry picked from commit 2bba276f0f06a5981bdd7e4f0e7e5ca2fe84f063)
2020-09-02 22:58:37 +03:00
function breeze::print_setup_instructions() {
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
if [[ ${command_to_run} == "enter_breeze" ]]; then
[AIRFLOW-5828] Move build logic out from hooks/build (#7618) This is the final step of simplifying the Breeze scripts by moving all the logic out from Travis' hooks/build
2020-03-07 18:09:06 +03:00
# shellcheck disable=SC2034 # Unused variables left for comp_breeze usage
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
if ! typeset -f "_comp_breeze" >/dev/null; then
Add packages to function names in bash (#10670) (#10696) Inspired by the Google Shell Guide where they mentioned separating package names with :: I realized that this was one of the missing pieces in the bash scripts of ours. While we already had packages (in libraries folders) it's been difficult to realise which function is where. With introducing packages - equal to the library file name we are *almost* at a level of a structured language - and it's easier to find the functions if you are looking for them. Way easier in fact. Part of #10576 (cherry picked from commit cc551ba793344800d2d396c13d7fd0c8eed97352) (cherry picked from commit 2bba276f0f06a5981bdd7e4f0e7e5ca2fe84f063)
2020-09-02 22:58:37 +03:00
breeze::print_line
[AIRFLOW-5828] Move build logic out from hooks/build (#7618) This is the final step of simplifying the Breeze scripts by moving all the logic out from Travis' hooks/build
2020-03-07 18:09:06 +03:00
echo
echo " You can setup autocomplete by running '${CMDNAME} setup-autocomplete'"
echo
echo
fi
Add packages to function names in bash (#10670) (#10696) Inspired by the Google Shell Guide where they mentioned separating package names with :: I realized that this was one of the missing pieces in the bash scripts of ours. While we already had packages (in libraries folders) it's been difficult to realise which function is where. With introducing packages - equal to the library file name we are *almost* at a level of a structured language - and it's easier to find the functions if you are looking for them. Way easier in fact. Part of #10576 (cherry picked from commit cc551ba793344800d2d396c13d7fd0c8eed97352) (cherry picked from commit 2bba276f0f06a5981bdd7e4f0e7e5ca2fe84f063)
2020-09-02 22:58:37 +03:00
breeze::print_line
[AIRFLOW-5828] Move build logic out from hooks/build (#7618) This is the final step of simplifying the Breeze scripts by moving all the logic out from Travis' hooks/build
2020-03-07 18:09:06 +03:00
echo
echo " You can toggle ascii/cheatsheet by running:"
echo " * ${CMDNAME} toggle-suppress-cheatsheet"
echo " * ${CMDNAME} toggle-suppress-asciiart"
echo
Add packages to function names in bash (#10670) (#10696) Inspired by the Google Shell Guide where they mentioned separating package names with :: I realized that this was one of the missing pieces in the bash scripts of ours. While we already had packages (in libraries folders) it's been difficult to realise which function is where. With introducing packages - equal to the library file name we are *almost* at a level of a structured language - and it's easier to find the functions if you are looking for them. Way easier in fact. Part of #10576 (cherry picked from commit cc551ba793344800d2d396c13d7fd0c8eed97352) (cherry picked from commit 2bba276f0f06a5981bdd7e4f0e7e5ca2fe84f063)
2020-09-02 22:58:37 +03:00
breeze::print_line
[AIRFLOW-6820] split breeze into functions (#7433)
2020-02-19 03:58:53 +03:00
echo
[AIRFLOW-5828] Move build logic out from hooks/build (#7618) This is the final step of simplifying the Breeze scripts by moving all the logic out from Travis' hooks/build
2020-03-07 18:09:06 +03:00
echo
[AIRFLOW-6820] split breeze into functions (#7433)
2020-02-19 03:58:53 +03:00
echo
echo
[AIRFLOW-3611] Simplified development environment (#4932)
2019-08-27 21:39:36 +03:00
fi
[AIRFLOW-6820] split breeze into functions (#7433)
2020-02-19 03:58:53 +03:00
}
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
#######################################################################################################
#
Add comments to breeze scripts (#8797)
2020-05-10 20:55:40 +03:00
# Checks that pre-commit is installed and upgrades it if needed
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
# this is used in case static check command is used.
#
#######################################################################################################
Add packages to function names in bash (#10670) (#10696) Inspired by the Google Shell Guide where they mentioned separating package names with :: I realized that this was one of the missing pieces in the bash scripts of ours. While we already had packages (in libraries folders) it's been difficult to realise which function is where. With introducing packages - equal to the library file name we are *almost* at a level of a structured language - and it's easier to find the functions if you are looking for them. Way easier in fact. Part of #10576 (cherry picked from commit cc551ba793344800d2d396c13d7fd0c8eed97352) (cherry picked from commit 2bba276f0f06a5981bdd7e4f0e7e5ca2fe84f063)
2020-09-02 22:58:37 +03:00
function breeze::make_sure_precommit_is_installed() {
[AIRFLOW-3611] Simplified development environment (#4932)
2019-08-27 21:39:36 +03:00
echo
[AIRFLOW-6820] split breeze into functions (#7433)
2020-02-19 03:58:53 +03:00
echo "Making sure pre-commit is installed"
echo
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
local pip_binary
[AIRFLOW-6820] split breeze into functions (#7433)
2020-02-19 03:58:53 +03:00
if command -v pip3 >/dev/null; then
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
pip_binary=pip3
[AIRFLOW-6820] split breeze into functions (#7433)
2020-02-19 03:58:53 +03:00
elif command -v pip >/dev/null; then
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
pip_binary=pip
[AIRFLOW-6820] split breeze into functions (#7433)
2020-02-19 03:58:53 +03:00
else
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
echo >&2
echo >&2 "ERROR: You need to have pip or pip3 in your PATH"
echo >&2
S
exit 1
[AIRFLOW-6820] split breeze into functions (#7433)
2020-02-19 03:58:53 +03:00
fi
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
"${pip_binary}" install --upgrade pre-commit >/dev/null 2>&1
[AIRFLOW-6820] split breeze into functions (#7433)
2020-02-19 03:58:53 +03:00
# Add ~/.local/bin to the path in case pip is run outside of virtualenv
export PATH="${PATH}":~/.local/bin
}
[AIRFLOW-3611] Simplified development environment (#4932)
2019-08-27 21:39:36 +03:00
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
#######################################################################################################
#
Add comments to breeze scripts (#8797)
2020-05-10 20:55:40 +03:00
# Removes CI and PROD images and cleans up the flag that indicates that the image was already built
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
#
# Used global constants:
# PYTHON_BASE_IMAGE
# AIRFLOW_CI_IMAGE
# AIRFLOW_PROD_IMAGE
# BUILT_CI_IMAGE_FLAG_FILE
#
#######################################################################################################
Add packages to function names in bash (#10670) (#10696) Inspired by the Google Shell Guide where they mentioned separating package names with :: I realized that this was one of the missing pieces in the bash scripts of ours. While we already had packages (in libraries folders) it's been difficult to realise which function is where. With introducing packages - equal to the library file name we are *almost* at a level of a structured language - and it's easier to find the functions if you are looking for them. Way easier in fact. Part of #10576 (cherry picked from commit cc551ba793344800d2d396c13d7fd0c8eed97352) (cherry picked from commit 2bba276f0f06a5981bdd7e4f0e7e5ca2fe84f063)
2020-09-02 22:58:37 +03:00
function breeze::remove_images() {
The verbose functions will not exit immediately if not asked to (#10731) The docker(), helm(), kubectl() functions replace the real tools to get verbose behaviour (we can print the exact command being executed for those. But when 'set +e' was set before the command was called - indicating that error in those functions should be ignored - this did not happen. The functions set 'set -e' just before returning the non-zero value, effectively exiting the script right after. This caused first time experience to be not good. The fix also fixes behaviour of stdout and stderr for those functions - previously they were joined to be able to be printed to OUTPUT_FILE but this lost the stderr/stdout distinction. Now both stdout and stderr are printed to the output file but they are also redirected to stdout/stderr respectively, so that 2>/dev/null works as expected. While fixing it, it turned out that one of the remove_images methods was not used any more - merged it with the breeze one.
2020-09-06 20:56:35 +03:00
set +e
docker rmi "${PYTHON_BASE_IMAGE}" 2>/dev/null >/dev/null
docker rmi "${AIRFLOW_CI_IMAGE}" 2>/dev/null >/dev/null
docker rmi "${AIRFLOW_PROD_IMAGE}" 2>/dev/null >/dev/null
set -e
echo
echo "###################################################################"
echo "NOTE!! Removed Airflow images for Python version ${PYTHON_MAJOR_MINOR_VERSION}."
echo " But the disk space in docker will be reclaimed only after"
echo " running 'docker system prune' command."
echo "###################################################################"
echo
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
rm -f "${BUILT_CI_IMAGE_FLAG_FILE}"
[AIRFLOW-6820] split breeze into functions (#7433)
2020-02-19 03:58:53 +03:00
}
[AIRFLOW-3611] Simplified development environment (#4932)
2019-08-27 21:39:36 +03:00
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
#######################################################################################################
#
Add comments to breeze scripts (#8797)
2020-05-10 20:55:40 +03:00
# Runs chosen static checks
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
#
# Uses variables:
# static_check
#
#######################################################################################################
Add packages to function names in bash (#10670) (#10696) Inspired by the Google Shell Guide where they mentioned separating package names with :: I realized that this was one of the missing pieces in the bash scripts of ours. While we already had packages (in libraries folders) it's been difficult to realise which function is where. With introducing packages - equal to the library file name we are *almost* at a level of a structured language - and it's easier to find the functions if you are looking for them. Way easier in fact. Part of #10576 (cherry picked from commit cc551ba793344800d2d396c13d7fd0c8eed97352) (cherry picked from commit 2bba276f0f06a5981bdd7e4f0e7e5ca2fe84f063)
2020-09-02 22:58:37 +03:00
function breeze::run_static_checks() {
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
if [[ ${static_check} == "all" ]]; then
[AIRFLOW-6820] split breeze into functions (#7433)
2020-02-19 03:58:53 +03:00
echo
echo "Running: pre-commit run" "${EXTRA_STATIC_CHECK_OPTIONS[@]}" "$@"
echo
pre-commit run "${EXTRA_STATIC_CHECK_OPTIONS[@]}" "$@"
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
elif [[ ${static_check} == "all-but-pylint" ]]; then
[AIRFLOW-6820] split breeze into functions (#7433)
2020-02-19 03:58:53 +03:00
echo
echo "Setting SKIP=pylint. Running: pre-commit run" "${EXTRA_STATIC_CHECK_OPTIONS[@]}" "$@"
echo
echo
SKIP=pylint pre-commit run "${EXTRA_STATIC_CHECK_OPTIONS[@]}" "$@"
else
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
if [[ ${static_check} == *"pylint"* || ${static_check} == *"mypy"* || ${static_check} == *"flake8"* ]]; then
echo
echo "Running build pre-commit before running the requested static check"
echo
pre-commit run "build"
fi
[AIRFLOW-6820] split breeze into functions (#7433)
2020-02-19 03:58:53 +03:00
echo
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
echo "Running: pre-commit run" "${static_check}" "${EXTRA_STATIC_CHECK_OPTIONS[@]}" "$@"
[AIRFLOW-6820] split breeze into functions (#7433)
2020-02-19 03:58:53 +03:00
echo
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
pre-commit run "${static_check}" "${EXTRA_STATIC_CHECK_OPTIONS[@]}" "$@"
[AIRFLOW-6820] split breeze into functions (#7433)
2020-02-19 03:58:53 +03:00
fi
}
[AIRFLOW-3611] Simplified development environment (#4932)
2019-08-27 21:39:36 +03:00
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
#######################################################################################################
#
# Runs Build before a command if needed - the right build command will be determined and used
# depending on which command you are trying ton
#
# Used global constants:
# PRODUCTION_IMAGE
# KIND_CLUSTER_OPERATION
# FORMATTED_KIND_OPERATIONS
#
# Used variables:
# command_to_run
#
#######################################################################################################
Add packages to function names in bash (#10670) (#10696) Inspired by the Google Shell Guide where they mentioned separating package names with :: I realized that this was one of the missing pieces in the bash scripts of ours. While we already had packages (in libraries folders) it's been difficult to realise which function is where. With introducing packages - equal to the library file name we are *almost* at a level of a structured language - and it's easier to find the functions if you are looking for them. Way easier in fact. Part of #10576 (cherry picked from commit cc551ba793344800d2d396c13d7fd0c8eed97352) (cherry picked from commit 2bba276f0f06a5981bdd7e4f0e7e5ca2fe84f063)
2020-09-02 22:58:37 +03:00
function breeze::run_build_command() {
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
case "${command_to_run}" in
run_tests | run_docker_compose)
Add packages to function names in bash (#10670) (#10696) Inspired by the Google Shell Guide where they mentioned separating package names with :: I realized that this was one of the missing pieces in the bash scripts of ours. While we already had packages (in libraries folders) it's been difficult to realise which function is where. With introducing packages - equal to the library file name we are *almost* at a level of a structured language - and it's easier to find the functions if you are looking for them. Way easier in fact. Part of #10576 (cherry picked from commit cc551ba793344800d2d396c13d7fd0c8eed97352) (cherry picked from commit 2bba276f0f06a5981bdd7e4f0e7e5ca2fe84f063)
2020-09-02 22:58:37 +03:00
build_images::prepare_ci_build
build_images::rebuild_ci_image_if_needed
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
;;
enter_breeze)
if [[ ${PRODUCTION_IMAGE} == "true" ]]; then
Add packages to function names in bash (#10670) (#10696) Inspired by the Google Shell Guide where they mentioned separating package names with :: I realized that this was one of the missing pieces in the bash scripts of ours. While we already had packages (in libraries folders) it's been difficult to realise which function is where. With introducing packages - equal to the library file name we are *almost* at a level of a structured language - and it's easier to find the functions if you are looking for them. Way easier in fact. Part of #10576 (cherry picked from commit cc551ba793344800d2d396c13d7fd0c8eed97352) (cherry picked from commit 2bba276f0f06a5981bdd7e4f0e7e5ca2fe84f063)
2020-09-02 22:58:37 +03:00
build_images::prepare_prod_build
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
else
Add packages to function names in bash (#10670) (#10696) Inspired by the Google Shell Guide where they mentioned separating package names with :: I realized that this was one of the missing pieces in the bash scripts of ours. While we already had packages (in libraries folders) it's been difficult to realise which function is where. With introducing packages - equal to the library file name we are *almost* at a level of a structured language - and it's easier to find the functions if you are looking for them. Way easier in fact. Part of #10576 (cherry picked from commit cc551ba793344800d2d396c13d7fd0c8eed97352) (cherry picked from commit 2bba276f0f06a5981bdd7e4f0e7e5ca2fe84f063)
2020-09-02 22:58:37 +03:00
build_images::prepare_ci_build
build_images::rebuild_ci_image_if_needed
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
fi
;;
build_docs | perform_static_checks | perform_generate_constraints | perform_prepare_backport_readme | perform_prepare_backport_packages)
Add packages to function names in bash (#10670) (#10696) Inspired by the Google Shell Guide where they mentioned separating package names with :: I realized that this was one of the missing pieces in the bash scripts of ours. While we already had packages (in libraries folders) it's been difficult to realise which function is where. With introducing packages - equal to the library file name we are *almost* at a level of a structured language - and it's easier to find the functions if you are looking for them. Way easier in fact. Part of #10576 (cherry picked from commit cc551ba793344800d2d396c13d7fd0c8eed97352) (cherry picked from commit 2bba276f0f06a5981bdd7e4f0e7e5ca2fe84f063)
2020-09-02 22:58:37 +03:00
build_images::prepare_ci_build
build_images::rebuild_ci_image_if_needed
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
;;
perform_push_image)
if [[ ${PRODUCTION_IMAGE} == "true" ]]; then
Add packages to function names in bash (#10670) (#10696) Inspired by the Google Shell Guide where they mentioned separating package names with :: I realized that this was one of the missing pieces in the bash scripts of ours. While we already had packages (in libraries folders) it's been difficult to realise which function is where. With introducing packages - equal to the library file name we are *almost* at a level of a structured language - and it's easier to find the functions if you are looking for them. Way easier in fact. Part of #10576 (cherry picked from commit cc551ba793344800d2d396c13d7fd0c8eed97352) (cherry picked from commit 2bba276f0f06a5981bdd7e4f0e7e5ca2fe84f063)
2020-09-02 22:58:37 +03:00
build_images::prepare_prod_build
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
else
Add packages to function names in bash (#10670) (#10696) Inspired by the Google Shell Guide where they mentioned separating package names with :: I realized that this was one of the missing pieces in the bash scripts of ours. While we already had packages (in libraries folders) it's been difficult to realise which function is where. With introducing packages - equal to the library file name we are *almost* at a level of a structured language - and it's easier to find the functions if you are looking for them. Way easier in fact. Part of #10576 (cherry picked from commit cc551ba793344800d2d396c13d7fd0c8eed97352) (cherry picked from commit 2bba276f0f06a5981bdd7e4f0e7e5ca2fe84f063)
2020-09-02 22:58:37 +03:00
build_images::prepare_ci_build
build_images::rebuild_ci_image_if_needed
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
fi
;;
build_image)
if [[ ${PRODUCTION_IMAGE} == "true" ]]; then
Add packages to function names in bash (#10670) (#10696) Inspired by the Google Shell Guide where they mentioned separating package names with :: I realized that this was one of the missing pieces in the bash scripts of ours. While we already had packages (in libraries folders) it's been difficult to realise which function is where. With introducing packages - equal to the library file name we are *almost* at a level of a structured language - and it's easier to find the functions if you are looking for them. Way easier in fact. Part of #10576 (cherry picked from commit cc551ba793344800d2d396c13d7fd0c8eed97352) (cherry picked from commit 2bba276f0f06a5981bdd7e4f0e7e5ca2fe84f063)
2020-09-02 22:58:37 +03:00
build_images::prepare_prod_build
build_images::build_prod_images
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
else
Add packages to function names in bash (#10670) (#10696) Inspired by the Google Shell Guide where they mentioned separating package names with :: I realized that this was one of the missing pieces in the bash scripts of ours. While we already had packages (in libraries folders) it's been difficult to realise which function is where. With introducing packages - equal to the library file name we are *almost* at a level of a structured language - and it's easier to find the functions if you are looking for them. Way easier in fact. Part of #10576 (cherry picked from commit cc551ba793344800d2d396c13d7fd0c8eed97352) (cherry picked from commit 2bba276f0f06a5981bdd7e4f0e7e5ca2fe84f063)
2020-09-02 22:58:37 +03:00
build_images::prepare_ci_build
md5sum::calculate_md5sum_for_all_files
build_images::build_ci_image
md5sum::update_all_md5
build_images::build_ci_image_manifest
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
fi
;;
cleanup_image | run_exec)
if [[ ${PRODUCTION_IMAGE} == "true" ]]; then
Add packages to function names in bash (#10670) (#10696) Inspired by the Google Shell Guide where they mentioned separating package names with :: I realized that this was one of the missing pieces in the bash scripts of ours. While we already had packages (in libraries folders) it's been difficult to realise which function is where. With introducing packages - equal to the library file name we are *almost* at a level of a structured language - and it's easier to find the functions if you are looking for them. Way easier in fact. Part of #10576 (cherry picked from commit cc551ba793344800d2d396c13d7fd0c8eed97352) (cherry picked from commit 2bba276f0f06a5981bdd7e4f0e7e5ca2fe84f063)
2020-09-02 22:58:37 +03:00
build_images::prepare_prod_build
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
else
Add packages to function names in bash (#10670) (#10696) Inspired by the Google Shell Guide where they mentioned separating package names with :: I realized that this was one of the missing pieces in the bash scripts of ours. While we already had packages (in libraries folders) it's been difficult to realise which function is where. With introducing packages - equal to the library file name we are *almost* at a level of a structured language - and it's easier to find the functions if you are looking for them. Way easier in fact. Part of #10576 (cherry picked from commit cc551ba793344800d2d396c13d7fd0c8eed97352) (cherry picked from commit 2bba276f0f06a5981bdd7e4f0e7e5ca2fe84f063)
2020-09-02 22:58:37 +03:00
build_images::prepare_ci_build
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
fi
;;
perform_initialize_local_virtualenv | perform_setup_autocomplete | toggle_suppress_cheatsheet | toggle_suppress_asciiart ) ;;
manage_kind_cluster)
if [[ ${KIND_CLUSTER_OPERATION} == "start" ]]; then
echo "Starts KinD cluster"
elif [[ ${KIND_CLUSTER_OPERATION} == "stop" ]]; then
echo "Stops KinD cluster"
elif [[ ${KIND_CLUSTER_OPERATION} == "restart" ]]; then
echo "Restarts KinD cluster"
elif [[ ${KIND_CLUSTER_OPERATION} == "recreate" ]]; then
echo "Recreates KinD cluster"
elif [[ ${KIND_CLUSTER_OPERATION} == "status" ]]; then
echo "Checks status of KinD cluster"
elif [[ ${KIND_CLUSTER_OPERATION} == "deploy" ]]; then
echo "Deploys Airflow to KinD cluster"
Add packages to function names in bash (#10670) (#10696) Inspired by the Google Shell Guide where they mentioned separating package names with :: I realized that this was one of the missing pieces in the bash scripts of ours. While we already had packages (in libraries folders) it's been difficult to realise which function is where. With introducing packages - equal to the library file name we are *almost* at a level of a structured language - and it's easier to find the functions if you are looking for them. Way easier in fact. Part of #10576 (cherry picked from commit cc551ba793344800d2d396c13d7fd0c8eed97352) (cherry picked from commit 2bba276f0f06a5981bdd7e4f0e7e5ca2fe84f063)
2020-09-02 22:58:37 +03:00
build_images::prepare_prod_build
build_images::build_prod_images
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
elif [[ ${KIND_CLUSTER_OPERATION} == "test" ]]; then
echo "Run Kubernetes tests with the KinD cluster "
elif [[ ${KIND_CLUSTER_OPERATION} == "shell" ]]; then
echo "Enter an interactive shell for kubernetes testing"
elif [[ -z ${KIND_CLUSTER_OPERATION=} ]]; then
echo
echo "Please provide an operation to run"
echo
echo "Should be one of:"
echo "${FORMATTED_KIND_OPERATIONS}"
echo
exit 1
else
echo
echo "ERROR: Unknown Kind Kubernetes cluster operation: '${KIND_CLUSTER_OPERATION}'"
echo
echo "Should be one of:"
echo "${FORMATTED_KIND_OPERATIONS}"
echo
exit 1
fi
;;
*)
echo >&2
echo >&2 "ERROR: Unknown command to run ${command_to_run}"
echo >&2
exit 1
;;
[AIRFLOW-6820] split breeze into functions (#7433)
2020-02-19 03:58:53 +03:00
esac
}
[AIRFLOW-3611] Simplified development environment (#4932)
2019-08-27 21:39:36 +03:00
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
#######################################################################################################
#
Add comments to breeze scripts (#8797)
2020-05-10 20:55:40 +03:00
# Runs the actual command - depending on the command chosen it will use the right
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
# Convenient script and run the right command for the script.
#
# Used variables:
# command_to_run
#
# Used global constants:
# PRODUCTION_IMAGE
# SCRIPTS_CI_DIR
# BUILD_CACHE_DIR
# KIND_CLUSTER_OPERATION
# EXTRA_DC_OPTIONS
#
# Set Global variables:
# RUN_TESTS
#######################################################################################################
Add packages to function names in bash (#10670) (#10696) Inspired by the Google Shell Guide where they mentioned separating package names with :: I realized that this was one of the missing pieces in the bash scripts of ours. While we already had packages (in libraries folders) it's been difficult to realise which function is where. With introducing packages - equal to the library file name we are *almost* at a level of a structured language - and it's easier to find the functions if you are looking for them. Way easier in fact. Part of #10576 (cherry picked from commit cc551ba793344800d2d396c13d7fd0c8eed97352) (cherry picked from commit 2bba276f0f06a5981bdd7e4f0e7e5ca2fe84f063)
2020-09-02 22:58:37 +03:00
function breeze::run_breeze_command() {
Fixes unbound variable on MacOS (#9335) Closes #9334
2020-06-16 21:25:24 +03:00
set +u
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
local dc_run_file
if [[ ${PRODUCTION_IMAGE} == "true" ]]; then
dc_run_file="${BUILD_CACHE_DIR}/${DOCKER_COMPOSE_RUN_SCRIPT_FOR_PROD}"
else
dc_run_file="${BUILD_CACHE_DIR}/${DOCKER_COMPOSE_RUN_SCRIPT_FOR_CI}"
fi
case "${command_to_run}" in
enter_breeze)
if [[ ${PRODUCTION_IMAGE} == "true" ]]; then
"${dc_run_file}" run --service-ports --rm airflow "${@}"
"${SCRIPTS_CI_DIR}/tools/ci_fix_ownership.sh"
else
"${dc_run_file}" run --service-ports --rm airflow "${@}"
"${SCRIPTS_CI_DIR}/tools/ci_fix_ownership.sh"
"${SCRIPTS_CI_DIR}/tools/ci_clear_tmp.sh"
fi
;;
run_exec)
# Unfortunately `docker-compose exec` does not support exec'ing into containers started with run :(
# so we have to find it manually
set +e
local airflow_testing_container
airflow_testing_container=$("${dc_run_file}" ps | grep airflow | awk '{print $1}' 2>/dev/null)
: "${airflow_testing_container:?"ERROR! Breeze must be running in order to exec into running container"}"
set -e
docker exec -it "${airflow_testing_container}" \
"/opt/airflow/scripts/in_container/entrypoint_exec.sh" "${@}"
;;
run_tests)
export RUN_TESTS="true"
readonly RUN_TESTS
"${BUILD_CACHE_DIR}/${DOCKER_COMPOSE_RUN_SCRIPT_FOR_CI}" run --service-ports --rm airflow "$@"
;;
run_docker_compose)
set +u
if [[ ${PRODUCTION_IMAGE} == "true" ]]; then
dc_run_file="${BUILD_CACHE_DIR}/${DOCKER_COMPOSE_RUN_SCRIPT_FOR_PROD}"
else
dc_run_file="${BUILD_CACHE_DIR}/${DOCKER_COMPOSE_RUN_SCRIPT_FOR_CI}"
fi
"${dc_run_file}" "${docker_compose_command}" "${EXTRA_DC_OPTIONS[@]}" "$@"
set -u
;;
perform_static_checks)
Add packages to function names in bash (#10670) (#10696) Inspired by the Google Shell Guide where they mentioned separating package names with :: I realized that this was one of the missing pieces in the bash scripts of ours. While we already had packages (in libraries folders) it's been difficult to realise which function is where. With introducing packages - equal to the library file name we are *almost* at a level of a structured language - and it's easier to find the functions if you are looking for them. Way easier in fact. Part of #10576 (cherry picked from commit cc551ba793344800d2d396c13d7fd0c8eed97352) (cherry picked from commit 2bba276f0f06a5981bdd7e4f0e7e5ca2fe84f063)
2020-09-02 22:58:37 +03:00
breeze::make_sure_precommit_is_installed
breeze::run_static_checks "${@}"
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
;;
build_image) ;;
cleanup_image)
Add packages to function names in bash (#10670) (#10696) Inspired by the Google Shell Guide where they mentioned separating package names with :: I realized that this was one of the missing pieces in the bash scripts of ours. While we already had packages (in libraries folders) it's been difficult to realise which function is where. With introducing packages - equal to the library file name we are *almost* at a level of a structured language - and it's easier to find the functions if you are looking for them. Way easier in fact. Part of #10576 (cherry picked from commit cc551ba793344800d2d396c13d7fd0c8eed97352) (cherry picked from commit 2bba276f0f06a5981bdd7e4f0e7e5ca2fe84f063)
2020-09-02 22:58:37 +03:00
breeze::remove_images
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
;;
perform_generate_constraints)
Add packages to function names in bash (#10670) (#10696) Inspired by the Google Shell Guide where they mentioned separating package names with :: I realized that this was one of the missing pieces in the bash scripts of ours. While we already had packages (in libraries folders) it's been difficult to realise which function is where. With introducing packages - equal to the library file name we are *almost* at a level of a structured language - and it's easier to find the functions if you are looking for them. Way easier in fact. Part of #10576 (cherry picked from commit cc551ba793344800d2d396c13d7fd0c8eed97352) (cherry picked from commit 2bba276f0f06a5981bdd7e4f0e7e5ca2fe84f063)
2020-09-02 22:58:37 +03:00
runs::run_generate_constraints
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
;;
perform_prepare_backport_packages)
Add packages to function names in bash (#10670) (#10696) Inspired by the Google Shell Guide where they mentioned separating package names with :: I realized that this was one of the missing pieces in the bash scripts of ours. While we already had packages (in libraries folders) it's been difficult to realise which function is where. With introducing packages - equal to the library file name we are *almost* at a level of a structured language - and it's easier to find the functions if you are looking for them. Way easier in fact. Part of #10576 (cherry picked from commit cc551ba793344800d2d396c13d7fd0c8eed97352) (cherry picked from commit 2bba276f0f06a5981bdd7e4f0e7e5ca2fe84f063)
2020-09-02 22:58:37 +03:00
runs::run_prepare_backport_packages "${@}"
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
;;
perform_prepare_backport_readme)
Add packages to function names in bash (#10670) (#10696) Inspired by the Google Shell Guide where they mentioned separating package names with :: I realized that this was one of the missing pieces in the bash scripts of ours. While we already had packages (in libraries folders) it's been difficult to realise which function is where. With introducing packages - equal to the library file name we are *almost* at a level of a structured language - and it's easier to find the functions if you are looking for them. Way easier in fact. Part of #10576 (cherry picked from commit cc551ba793344800d2d396c13d7fd0c8eed97352) (cherry picked from commit 2bba276f0f06a5981bdd7e4f0e7e5ca2fe84f063)
2020-09-02 22:58:37 +03:00
runs::run_prepare_backport_readme "${@}"
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
;;
perform_push_image)
if [[ ${PRODUCTION_IMAGE} == "true" ]]; then
Add packages to function names in bash (#10670) (#10696) Inspired by the Google Shell Guide where they mentioned separating package names with :: I realized that this was one of the missing pieces in the bash scripts of ours. While we already had packages (in libraries folders) it's been difficult to realise which function is where. With introducing packages - equal to the library file name we are *almost* at a level of a structured language - and it's easier to find the functions if you are looking for them. Way easier in fact. Part of #10576 (cherry picked from commit cc551ba793344800d2d396c13d7fd0c8eed97352) (cherry picked from commit 2bba276f0f06a5981bdd7e4f0e7e5ca2fe84f063)
2020-09-02 22:58:37 +03:00
push_pull_remove_images::push_prod_images
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
else
Add packages to function names in bash (#10670) (#10696) Inspired by the Google Shell Guide where they mentioned separating package names with :: I realized that this was one of the missing pieces in the bash scripts of ours. While we already had packages (in libraries folders) it's been difficult to realise which function is where. With introducing packages - equal to the library file name we are *almost* at a level of a structured language - and it's easier to find the functions if you are looking for them. Way easier in fact. Part of #10576 (cherry picked from commit cc551ba793344800d2d396c13d7fd0c8eed97352) (cherry picked from commit 2bba276f0f06a5981bdd7e4f0e7e5ca2fe84f063)
2020-09-02 22:58:37 +03:00
push_pull_remove_images::push_ci_images
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
fi
;;
perform_initialize_local_virtualenv)
Add packages to function names in bash (#10670) (#10696) Inspired by the Google Shell Guide where they mentioned separating package names with :: I realized that this was one of the missing pieces in the bash scripts of ours. While we already had packages (in libraries folders) it's been difficult to realise which function is where. With introducing packages - equal to the library file name we are *almost* at a level of a structured language - and it's easier to find the functions if you are looking for them. Way easier in fact. Part of #10576 (cherry picked from commit cc551ba793344800d2d396c13d7fd0c8eed97352) (cherry picked from commit 2bba276f0f06a5981bdd7e4f0e7e5ca2fe84f063)
2020-09-02 22:58:37 +03:00
breeze::initialize_virtualenv
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
;;
perform_setup_autocomplete)
Add packages to function names in bash (#10670) (#10696) Inspired by the Google Shell Guide where they mentioned separating package names with :: I realized that this was one of the missing pieces in the bash scripts of ours. While we already had packages (in libraries folders) it's been difficult to realise which function is where. With introducing packages - equal to the library file name we are *almost* at a level of a structured language - and it's easier to find the functions if you are looking for them. Way easier in fact. Part of #10576 (cherry picked from commit cc551ba793344800d2d396c13d7fd0c8eed97352) (cherry picked from commit 2bba276f0f06a5981bdd7e4f0e7e5ca2fe84f063)
2020-09-02 22:58:37 +03:00
breeze::setup_autocomplete
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
;;
manage_kind_cluster)
Add packages to function names in bash (#10670) (#10696) Inspired by the Google Shell Guide where they mentioned separating package names with :: I realized that this was one of the missing pieces in the bash scripts of ours. While we already had packages (in libraries folders) it's been difficult to realise which function is where. With introducing packages - equal to the library file name we are *almost* at a level of a structured language - and it's easier to find the functions if you are looking for them. Way easier in fact. Part of #10576 (cherry picked from commit cc551ba793344800d2d396c13d7fd0c8eed97352) (cherry picked from commit 2bba276f0f06a5981bdd7e4f0e7e5ca2fe84f063)
2020-09-02 22:58:37 +03:00
kind::make_sure_kubernetes_tools_are_installed
kind::get_kind_cluster_name
kind::perform_kind_cluster_operation "${KIND_CLUSTER_OPERATION}"
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
;;
build_docs)
Add packages to function names in bash (#10670) (#10696) Inspired by the Google Shell Guide where they mentioned separating package names with :: I realized that this was one of the missing pieces in the bash scripts of ours. While we already had packages (in libraries folders) it's been difficult to realise which function is where. With introducing packages - equal to the library file name we are *almost* at a level of a structured language - and it's easier to find the functions if you are looking for them. Way easier in fact. Part of #10576 (cherry picked from commit cc551ba793344800d2d396c13d7fd0c8eed97352) (cherry picked from commit 2bba276f0f06a5981bdd7e4f0e7e5ca2fe84f063)
2020-09-02 22:58:37 +03:00
runs::run_docs "${@}"
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
;;
toggle_suppress_cheatsheet)
if [[ -f "${SUPPRESS_CHEATSHEET_FILE}" ]]; then
echo
echo "Cheatsheet disabled"
echo
else
echo
echo "Cheatsheet enabled"
echo
fi
;;
toggle_suppress_asciiart)
if [[ -f "${SUPPRESS_ASCIIART_FILE}" ]]; then
echo
echo "ASCIIart disabled"
echo
else
echo
echo "ASCIIart enabled"
echo
fi
;;
*)
echo >&2
echo >&2 "ERROR: Unknown command to run ${command_to_run}"
echo >&2
;;
[AIRFLOW-6820] split breeze into functions (#7433)
2020-02-19 03:58:53 +03:00
esac
Fixes unbound variable on MacOS (#9335) Closes #9334
2020-06-16 21:25:24 +03:00
set -u
[AIRFLOW-6820] split breeze into functions (#7433)
2020-02-19 03:58:53 +03:00
}
[AIRFLOW-5704] Improve Kind Kubernetes scripts for local testing (#6516) * Fixed problem that Kubernetes tests were testing latest master rather than what came from the local sources. * Kind (Kubernetes in Dcocker) is run in the same Docker as Breeze env * Moved Kubernetes scripts to 'in_container' dir where they belong now * Kubernetes cluster is reused until it is stopped * Kubernetes image is build from image in docker already + mounted sources * Kubectl version name is corrected in the Dockerfile * KUBERNETES_VERSION can now be used to select Kubernetes version * Running kubernetes scripts is now easy in Breeze * We can start/recreate/stop cluster using --<ACTION>-kind-cluster * Instructions on how to run Kubernetes tests are updated * The old "bare" environment is replaced by --no-deps switch
2020-01-11 18:25:19 +03:00
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
#######################################################################################################
#
# We have different versions of images depending on the python version used. We keep up with the
# Latest patch-level changes in Python (this is done automatically during CI builds) so we have
# To only take into account MAJOR and MINOR version of python. This variable keeps the major/minor
# version of python in X.Y format (3.6, 3.7, 3.8 etc).
#
# In Breeze the precedence of setting the version is as follows:
# 1. --python flag (if set, it will explicitly override it in the next step)
# 2. PYTHON_MAJOR_MINOR_VERSION exported from outside
# 3. last used version stored in ./build/PYTHON_MAJOR_MINOR_VERSION
# 4. DEFAULT_PYTHON_MAJOR_MINOR_VERSION from scripts/ci/libraries/_initialization.sh
#
# Here points 2. and 3. are realised. If result is empty string , the 4. will be set in
Add packages to function names in bash (#10670) (#10696) Inspired by the Google Shell Guide where they mentioned separating package names with :: I realized that this was one of the missing pieces in the bash scripts of ours. While we already had packages (in libraries folders) it's been difficult to realise which function is where. With introducing packages - equal to the library file name we are *almost* at a level of a structured language - and it's easier to find the functions if you are looking for them. Way easier in fact. Part of #10576 (cherry picked from commit cc551ba793344800d2d396c13d7fd0c8eed97352) (cherry picked from commit 2bba276f0f06a5981bdd7e4f0e7e5ca2fe84f063)
2020-09-02 22:58:37 +03:00
# the next step (sanity_checks::basic_sanity_checks() is called and the version is still not set by then)
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
# finally, if --python flag is specified, it will override whatever is set above.
#
Add packages to function names in bash (#10670) (#10696) Inspired by the Google Shell Guide where they mentioned separating package names with :: I realized that this was one of the missing pieces in the bash scripts of ours. While we already had packages (in libraries folders) it's been difficult to realise which function is where. With introducing packages - equal to the library file name we are *almost* at a level of a structured language - and it's easier to find the functions if you are looking for them. Way easier in fact. Part of #10576 (cherry picked from commit cc551ba793344800d2d396c13d7fd0c8eed97352) (cherry picked from commit 2bba276f0f06a5981bdd7e4f0e7e5ca2fe84f063)
2020-09-02 22:58:37 +03:00
# We need to run after initialization::initialize_common_environment (so that parameters::read_from_file function is present)
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
# But before we set the default value for Python
#
# Used and modified global constants:
# PYTHON_MAJOR_MINOR_VERSION
#######################################################################################################
Add packages to function names in bash (#10670) (#10696) Inspired by the Google Shell Guide where they mentioned separating package names with :: I realized that this was one of the missing pieces in the bash scripts of ours. While we already had packages (in libraries folders) it's been difficult to realise which function is where. With introducing packages - equal to the library file name we are *almost* at a level of a structured language - and it's easier to find the functions if you are looking for them. Way easier in fact. Part of #10576 (cherry picked from commit cc551ba793344800d2d396c13d7fd0c8eed97352) (cherry picked from commit 2bba276f0f06a5981bdd7e4f0e7e5ca2fe84f063)
2020-09-02 22:58:37 +03:00
function breeze::determine_python_version_to_use_in_breeze() {
PYTHON_MAJOR_MINOR_VERSION="${PYTHON_MAJOR_MINOR_VERSION:=$(parameters::read_from_file PYTHON_MAJOR_MINOR_VERSION)}"
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
export PYTHON_MAJOR_MINOR_VERSION
}
Add packages to function names in bash (#10670) (#10696) Inspired by the Google Shell Guide where they mentioned separating package names with :: I realized that this was one of the missing pieces in the bash scripts of ours. While we already had packages (in libraries folders) it's been difficult to realise which function is where. With introducing packages - equal to the library file name we are *almost* at a level of a structured language - and it's easier to find the functions if you are looking for them. Way easier in fact. Part of #10576 (cherry picked from commit cc551ba793344800d2d396c13d7fd0c8eed97352) (cherry picked from commit 2bba276f0f06a5981bdd7e4f0e7e5ca2fe84f063)
2020-09-02 22:58:37 +03:00
breeze::setup_default_breeze_constants
[AIRFLOW-6489] Separate integrations in tests (#7091) You can now choose which integration you want to start when you run Breeze as well as when CI tests are run. Now by default Breeze and CI runs without integrations, but you can add them via Breeze flags or by environment variables when CI is executed. We have pytest markers now that mark tests that can be run for integration, backend and runtime selected. Also we have now more test jobs - we have separate test run for all non-integration tests (with less memory used by the integrations) and separate jobs that run integration tests only (more memory used for integrations but far less number of tests to run)
2020-01-13 22:47:37 +03:00
Add packages to function names in bash (#10670) (#10696) Inspired by the Google Shell Guide where they mentioned separating package names with :: I realized that this was one of the missing pieces in the bash scripts of ours. While we already had packages (in libraries folders) it's been difficult to realise which function is where. With introducing packages - equal to the library file name we are *almost* at a level of a structured language - and it's easier to find the functions if you are looking for them. Way easier in fact. Part of #10576 (cherry picked from commit cc551ba793344800d2d396c13d7fd0c8eed97352) (cherry picked from commit 2bba276f0f06a5981bdd7e4f0e7e5ca2fe84f063)
2020-09-02 22:58:37 +03:00
initialization::initialize_common_environment
[AIRFLOW-6489] Separate integrations in tests (#7091) You can now choose which integration you want to start when you run Breeze as well as when CI tests are run. Now by default Breeze and CI runs without integrations, but you can add them via Breeze flags or by environment variables when CI is executed. We have pytest markers now that mark tests that can be run for integration, backend and runtime selected. Also we have now more test jobs - we have separate test run for all non-integration tests (with less memory used by the integrations) and separate jobs that run integration tests only (more memory used for integrations but far less number of tests to run)
2020-01-13 22:47:37 +03:00
Add packages to function names in bash (#10670) (#10696) Inspired by the Google Shell Guide where they mentioned separating package names with :: I realized that this was one of the missing pieces in the bash scripts of ours. While we already had packages (in libraries folders) it's been difficult to realise which function is where. With introducing packages - equal to the library file name we are *almost* at a level of a structured language - and it's easier to find the functions if you are looking for them. Way easier in fact. Part of #10576 (cherry picked from commit cc551ba793344800d2d396c13d7fd0c8eed97352) (cherry picked from commit 2bba276f0f06a5981bdd7e4f0e7e5ca2fe84f063)
2020-09-02 22:58:37 +03:00
breeze::determine_python_version_to_use_in_breeze
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
Add packages to function names in bash (#10670) (#10696) Inspired by the Google Shell Guide where they mentioned separating package names with :: I realized that this was one of the missing pieces in the bash scripts of ours. While we already had packages (in libraries folders) it's been difficult to realise which function is where. With introducing packages - equal to the library file name we are *almost* at a level of a structured language - and it's easier to find the functions if you are looking for them. Way easier in fact. Part of #10576 (cherry picked from commit cc551ba793344800d2d396c13d7fd0c8eed97352) (cherry picked from commit 2bba276f0f06a5981bdd7e4f0e7e5ca2fe84f063)
2020-09-02 22:58:37 +03:00
sanity_checks::basic_sanity_checks
Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals.
2020-08-31 14:24:53 +03:00
Add packages to function names in bash (#10670) (#10696) Inspired by the Google Shell Guide where they mentioned separating package names with :: I realized that this was one of the missing pieces in the bash scripts of ours. While we already had packages (in libraries folders) it's been difficult to realise which function is where. With introducing packages - equal to the library file name we are *almost* at a level of a structured language - and it's easier to find the functions if you are looking for them. Way easier in fact. Part of #10576 (cherry picked from commit cc551ba793344800d2d396c13d7fd0c8eed97352) (cherry picked from commit 2bba276f0f06a5981bdd7e4f0e7e5ca2fe84f063)
2020-09-02 22:58:37 +03:00
start_end::script_start
[AIRFLOW-3611] Simplified development environment (#4932)
2019-08-27 21:39:36 +03:00
Add packages to function names in bash (#10670) (#10696) Inspired by the Google Shell Guide where they mentioned separating package names with :: I realized that this was one of the missing pieces in the bash scripts of ours. While we already had packages (in libraries folders) it's been difficult to realise which function is where. With introducing packages - equal to the library file name we are *almost* at a level of a structured language - and it's easier to find the functions if you are looking for them. Way easier in fact. Part of #10576 (cherry picked from commit cc551ba793344800d2d396c13d7fd0c8eed97352) (cherry picked from commit 2bba276f0f06a5981bdd7e4f0e7e5ca2fe84f063)
2020-09-02 22:58:37 +03:00
traps::add_trap start_end::script_end EXIT
[AIRFLOW-3611] Simplified development environment (#4932)
2019-08-27 21:39:36 +03:00
Add packages to function names in bash (#10670) (#10696) Inspired by the Google Shell Guide where they mentioned separating package names with :: I realized that this was one of the missing pieces in the bash scripts of ours. While we already had packages (in libraries folders) it's been difficult to realise which function is where. With introducing packages - equal to the library file name we are *almost* at a level of a structured language - and it's easier to find the functions if you are looking for them. Way easier in fact. Part of #10576 (cherry picked from commit cc551ba793344800d2d396c13d7fd0c8eed97352) (cherry picked from commit 2bba276f0f06a5981bdd7e4f0e7e5ca2fe84f063)
2020-09-02 22:58:37 +03:00
breeze::prepare_formatted_versions
[AIRFLOW-5704] Improve Kind Kubernetes scripts for local testing (#6516) * Fixed problem that Kubernetes tests were testing latest master rather than what came from the local sources. * Kind (Kubernetes in Dcocker) is run in the same Docker as Breeze env * Moved Kubernetes scripts to 'in_container' dir where they belong now * Kubernetes cluster is reused until it is stopped * Kubernetes image is build from image in docker already + mounted sources * Kubectl version name is corrected in the Dockerfile * KUBERNETES_VERSION can now be used to select Kubernetes version * Running kubernetes scripts is now easy in Breeze * We can start/recreate/stop cluster using --<ACTION>-kind-cluster * Instructions on how to run Kubernetes tests are updated * The old "bare" environment is replaced by --no-deps switch
2020-01-11 18:25:19 +03:00
Add packages to function names in bash (#10670) (#10696) Inspired by the Google Shell Guide where they mentioned separating package names with :: I realized that this was one of the missing pieces in the bash scripts of ours. While we already had packages (in libraries folders) it's been difficult to realise which function is where. With introducing packages - equal to the library file name we are *almost* at a level of a structured language - and it's easier to find the functions if you are looking for them. Way easier in fact. Part of #10576 (cherry picked from commit cc551ba793344800d2d396c13d7fd0c8eed97352) (cherry picked from commit 2bba276f0f06a5981bdd7e4f0e7e5ca2fe84f063)
2020-09-02 22:58:37 +03:00
breeze::prepare_usage
[AIRFLOW-6838] Introduce real subcommands for Breeze (#7515) This change introduces sub-commands in breeze tool. It is much needed as we have many commands now and it was difficult to separate commands from flags. Also --help output was very long and unreadable. With this change help it is much easier to discover what breeze can do for you as well as navigate with it. Co-authored-by: Jarek Potiuk <jarek@potiuk.com> Co-authored-by: Kamil Breguła <mik-laj@users.noreply.github.com>
2020-02-25 00:31:50 +03:00
[AIRFLOW-6841] Fixed unbounded variable on Mac (#7465)
2020-02-19 19:35:15 +03:00
set +u
Add packages to function names in bash (#10670) (#10696) Inspired by the Google Shell Guide where they mentioned separating package names with :: I realized that this was one of the missing pieces in the bash scripts of ours. While we already had packages (in libraries folders) it's been difficult to realise which function is where. With introducing packages - equal to the library file name we are *almost* at a level of a structured language - and it's easier to find the functions if you are looking for them. Way easier in fact. Part of #10576 (cherry picked from commit cc551ba793344800d2d396c13d7fd0c8eed97352) (cherry picked from commit 2bba276f0f06a5981bdd7e4f0e7e5ca2fe84f063)
2020-09-02 22:58:37 +03:00
breeze::parse_arguments "${@}"
[AIRFLOW-3611] Simplified development environment (#4932)
2019-08-27 21:39:36 +03:00
Add packages to function names in bash (#10670) (#10696) Inspired by the Google Shell Guide where they mentioned separating package names with :: I realized that this was one of the missing pieces in the bash scripts of ours. While we already had packages (in libraries folders) it's been difficult to realise which function is where. With introducing packages - equal to the library file name we are *almost* at a level of a structured language - and it's easier to find the functions if you are looking for them. Way easier in fact. Part of #10576 (cherry picked from commit cc551ba793344800d2d396c13d7fd0c8eed97352) (cherry picked from commit 2bba276f0f06a5981bdd7e4f0e7e5ca2fe84f063)
2020-09-02 22:58:37 +03:00
breeze::print_header_line
[AIRFLOW-3611] Simplified development environment (#4932)
2019-08-27 21:39:36 +03:00
Add packages to function names in bash (#10670) (#10696) Inspired by the Google Shell Guide where they mentioned separating package names with :: I realized that this was one of the missing pieces in the bash scripts of ours. While we already had packages (in libraries folders) it's been difficult to realise which function is where. With introducing packages - equal to the library file name we are *almost* at a level of a structured language - and it's easier to find the functions if you are looking for them. Way easier in fact. Part of #10576 (cherry picked from commit cc551ba793344800d2d396c13d7fd0c8eed97352) (cherry picked from commit 2bba276f0f06a5981bdd7e4f0e7e5ca2fe84f063)
2020-09-02 22:58:37 +03:00
build_images::forget_last_answer
Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals.
2020-08-31 14:24:53 +03:00
Add packages to function names in bash (#10670) (#10696) Inspired by the Google Shell Guide where they mentioned separating package names with :: I realized that this was one of the missing pieces in the bash scripts of ours. While we already had packages (in libraries folders) it's been difficult to realise which function is where. With introducing packages - equal to the library file name we are *almost* at a level of a structured language - and it's easier to find the functions if you are looking for them. Way easier in fact. Part of #10576 (cherry picked from commit cc551ba793344800d2d396c13d7fd0c8eed97352) (cherry picked from commit 2bba276f0f06a5981bdd7e4f0e7e5ca2fe84f063)
2020-09-02 22:58:37 +03:00
breeze::read_saved_environment_variables
Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals.
2020-08-31 14:24:53 +03:00
Add packages to function names in bash (#10670) (#10696) Inspired by the Google Shell Guide where they mentioned separating package names with :: I realized that this was one of the missing pieces in the bash scripts of ours. While we already had packages (in libraries folders) it's been difficult to realise which function is where. With introducing packages - equal to the library file name we are *almost* at a level of a structured language - and it's easier to find the functions if you are looking for them. Way easier in fact. Part of #10576 (cherry picked from commit cc551ba793344800d2d396c13d7fd0c8eed97352) (cherry picked from commit 2bba276f0f06a5981bdd7e4f0e7e5ca2fe84f063)
2020-09-02 22:58:37 +03:00
breeze::check_and_save_all_params
Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals.
2020-08-31 14:24:53 +03:00
Add packages to function names in bash (#10670) (#10696) Inspired by the Google Shell Guide where they mentioned separating package names with :: I realized that this was one of the missing pieces in the bash scripts of ours. While we already had packages (in libraries folders) it's been difficult to realise which function is where. With introducing packages - equal to the library file name we are *almost* at a level of a structured language - and it's easier to find the functions if you are looking for them. Way easier in fact. Part of #10576 (cherry picked from commit cc551ba793344800d2d396c13d7fd0c8eed97352) (cherry picked from commit 2bba276f0f06a5981bdd7e4f0e7e5ca2fe84f063)
2020-09-02 22:58:37 +03:00
build_images::determine_docker_cache_strategy
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
Add packages to function names in bash (#10670) (#10696) Inspired by the Google Shell Guide where they mentioned separating package names with :: I realized that this was one of the missing pieces in the bash scripts of ours. While we already had packages (in libraries folders) it's been difficult to realise which function is where. With introducing packages - equal to the library file name we are *almost* at a level of a structured language - and it's easier to find the functions if you are looking for them. Way easier in fact. Part of #10576 (cherry picked from commit cc551ba793344800d2d396c13d7fd0c8eed97352) (cherry picked from commit 2bba276f0f06a5981bdd7e4f0e7e5ca2fe84f063)
2020-09-02 22:58:37 +03:00
build_images::get_docker_image_names
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
Add packages to function names in bash (#10670) (#10696) Inspired by the Google Shell Guide where they mentioned separating package names with :: I realized that this was one of the missing pieces in the bash scripts of ours. While we already had packages (in libraries folders) it's been difficult to realise which function is where. With introducing packages - equal to the library file name we are *almost* at a level of a structured language - and it's easier to find the functions if you are looking for them. Way easier in fact. Part of #10576 (cherry picked from commit cc551ba793344800d2d396c13d7fd0c8eed97352) (cherry picked from commit 2bba276f0f06a5981bdd7e4f0e7e5ca2fe84f063)
2020-09-02 22:58:37 +03:00
initialization::make_constants_read_only
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
Add packages to function names in bash (#10670) (#10696) Inspired by the Google Shell Guide where they mentioned separating package names with :: I realized that this was one of the missing pieces in the bash scripts of ours. While we already had packages (in libraries folders) it's been difficult to realise which function is where. With introducing packages - equal to the library file name we are *almost* at a level of a structured language - and it's easier to find the functions if you are looking for them. Way easier in fact. Part of #10576 (cherry picked from commit cc551ba793344800d2d396c13d7fd0c8eed97352) (cherry picked from commit 2bba276f0f06a5981bdd7e4f0e7e5ca2fe84f063)
2020-09-02 22:58:37 +03:00
sanity_checks::sanitize_mounted_files
[AIRFLOW-3611] Simplified development environment (#4932)
2019-08-27 21:39:36 +03:00
Add packages to function names in bash (#10670) (#10696) Inspired by the Google Shell Guide where they mentioned separating package names with :: I realized that this was one of the missing pieces in the bash scripts of ours. While we already had packages (in libraries folders) it's been difficult to realise which function is where. With introducing packages - equal to the library file name we are *almost* at a level of a structured language - and it's easier to find the functions if you are looking for them. Way easier in fact. Part of #10576 (cherry picked from commit cc551ba793344800d2d396c13d7fd0c8eed97352) (cherry picked from commit 2bba276f0f06a5981bdd7e4f0e7e5ca2fe84f063)
2020-09-02 22:58:37 +03:00
breeze::prepare_command_files
[AIRFLOW-3611] Simplified development environment (#4932)
2019-08-27 21:39:36 +03:00
Add packages to function names in bash (#10670) (#10696) Inspired by the Google Shell Guide where they mentioned separating package names with :: I realized that this was one of the missing pieces in the bash scripts of ours. While we already had packages (in libraries folders) it's been difficult to realise which function is where. With introducing packages - equal to the library file name we are *almost* at a level of a structured language - and it's easier to find the functions if you are looking for them. Way easier in fact. Part of #10576 (cherry picked from commit cc551ba793344800d2d396c13d7fd0c8eed97352) (cherry picked from commit 2bba276f0f06a5981bdd7e4f0e7e5ca2fe84f063)
2020-09-02 22:58:37 +03:00
breeze::run_build_command
[AIRFLOW-3611] Simplified development environment (#4932)
2019-08-27 21:39:36 +03:00
Add packages to function names in bash (#10670) (#10696) Inspired by the Google Shell Guide where they mentioned separating package names with :: I realized that this was one of the missing pieces in the bash scripts of ours. While we already had packages (in libraries folders) it's been difficult to realise which function is where. With introducing packages - equal to the library file name we are *almost* at a level of a structured language - and it's easier to find the functions if you are looking for them. Way easier in fact. Part of #10576 (cherry picked from commit cc551ba793344800d2d396c13d7fd0c8eed97352) (cherry picked from commit 2bba276f0f06a5981bdd7e4f0e7e5ca2fe84f063)
2020-09-02 22:58:37 +03:00
breeze::print_header_line
[AIRFLOW-3611] Simplified development environment (#4932)
2019-08-27 21:39:36 +03:00
Add packages to function names in bash (#10670) (#10696) Inspired by the Google Shell Guide where they mentioned separating package names with :: I realized that this was one of the missing pieces in the bash scripts of ours. While we already had packages (in libraries folders) it's been difficult to realise which function is where. With introducing packages - equal to the library file name we are *almost* at a level of a structured language - and it's easier to find the functions if you are looking for them. Way easier in fact. Part of #10576 (cherry picked from commit cc551ba793344800d2d396c13d7fd0c8eed97352) (cherry picked from commit 2bba276f0f06a5981bdd7e4f0e7e5ca2fe84f063)
2020-09-02 22:58:37 +03:00
breeze::print_badge
[AIRFLOW-3611] Simplified development environment (#4932)
2019-08-27 21:39:36 +03:00
Add packages to function names in bash (#10670) (#10696) Inspired by the Google Shell Guide where they mentioned separating package names with :: I realized that this was one of the missing pieces in the bash scripts of ours. While we already had packages (in libraries folders) it's been difficult to realise which function is where. With introducing packages - equal to the library file name we are *almost* at a level of a structured language - and it's easier to find the functions if you are looking for them. Way easier in fact. Part of #10576 (cherry picked from commit cc551ba793344800d2d396c13d7fd0c8eed97352) (cherry picked from commit 2bba276f0f06a5981bdd7e4f0e7e5ca2fe84f063)
2020-09-02 22:58:37 +03:00
breeze::print_cheatsheet
[AIRFLOW-3611] Simplified development environment (#4932)
2019-08-27 21:39:36 +03:00
Add packages to function names in bash (#10670) (#10696) Inspired by the Google Shell Guide where they mentioned separating package names with :: I realized that this was one of the missing pieces in the bash scripts of ours. While we already had packages (in libraries folders) it's been difficult to realise which function is where. With introducing packages - equal to the library file name we are *almost* at a level of a structured language - and it's easier to find the functions if you are looking for them. Way easier in fact. Part of #10576 (cherry picked from commit cc551ba793344800d2d396c13d7fd0c8eed97352) (cherry picked from commit 2bba276f0f06a5981bdd7e4f0e7e5ca2fe84f063)
2020-09-02 22:58:37 +03:00
breeze::print_setup_instructions
[AIRFLOW-3611] Simplified development environment (#4932)
2019-08-27 21:39:36 +03:00
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
set +u # Account for an empty array
Add packages to function names in bash (#10670) (#10696) Inspired by the Google Shell Guide where they mentioned separating package names with :: I realized that this was one of the missing pieces in the bash scripts of ours. While we already had packages (in libraries folders) it's been difficult to realise which function is where. With introducing packages - equal to the library file name we are *almost* at a level of a structured language - and it's easier to find the functions if you are looking for them. Way easier in fact. Part of #10576 (cherry picked from commit cc551ba793344800d2d396c13d7fd0c8eed97352) (cherry picked from commit 2bba276f0f06a5981bdd7e4f0e7e5ca2fe84f063)
2020-09-02 22:58:37 +03:00
breeze::run_breeze_command "${REMAINING_ARGS[@]}"
[AIRFLOW-6932] Add restart-environment command to Breeze (#7557) When you switch between versions of Aiflow installed, you want to delete the database so that the scripts for resetdb work
2020-02-27 12:44:43 +03:00
Implement Google Shell Conventions for breeze script (#10695) * Implement Google Shell Conventions for breeze script … (#10651) Part of #10576 First (and the biggest of the series of commits to introduce Google Shell Conventions in our bash scripts. This is about the biggest and the most complex breeze script so it is rather huge but it is difficult to split it into smaller pieces. The rules implemented (from the conventions): * constants and exported variables are CAPITALIZED, where local/temporary variables are lowercase * following the shell guide, once all the variables are set to their final values (either from exported variables, calculation or --switches ) I have a single function that makes all the variables read-only. That helped to clean-up a lot of places where same functions was called several times, or where variables were defined in a few places. Now the behavior should be rather consistent and we should easily catch some duplications * function headers (following the guide) explaining arguments, variables expected, variables modified in the functions used. * setting the variables as read-only also helped to clean-up the "ifs" where we often had ":=}" in variables and != "" or == "". Those are replaced with `=}` and tests are replaced with `-n` and `-z` - also following the shell guide (readonly helped to detect and clean all such cases). This also should be much more robust in the future. * reorganized initialization of those constants and variables - simplified a few places where initialization was overlapping. It should be much more straightforward and clean now * a number of internal function breeze variables are "local" - this is helpful in accidental variables overwriting and keeping stuff localized * trap_add function is separated out to help in cases where we had several traps handling the same signals. (cherry picked from commit 46c8d6714c981746cc114b8b1af5cb27aa0018e2) (cherry picked from commit c822fd7b4bf2a9c5a9bb3c6e783cbea9dac37246) * fixup! Implement Google Shell Conventions for breeze script … (#10651)
2020-09-02 22:55:50 +03:00
set +u # Account for an empty array
if [[ -n ${second_command_to_run} ]]; then
command_to_run=${second_command_to_run}
Add packages to function names in bash (#10670) (#10696) Inspired by the Google Shell Guide where they mentioned separating package names with :: I realized that this was one of the missing pieces in the bash scripts of ours. While we already had packages (in libraries folders) it's been difficult to realise which function is where. With introducing packages - equal to the library file name we are *almost* at a level of a structured language - and it's easier to find the functions if you are looking for them. Way easier in fact. Part of #10576 (cherry picked from commit cc551ba793344800d2d396c13d7fd0c8eed97352) (cherry picked from commit 2bba276f0f06a5981bdd7e4f0e7e5ca2fe84f063)
2020-09-02 22:58:37 +03:00
breeze::run_breeze_command "${REMAINING_ARGS[@]}"
[AIRFLOW-6932] Add restart-environment command to Breeze (#7557) When you switch between versions of Aiflow installed, you want to delete the database so that the scripts for resetdb work
2020-02-27 12:44:43 +03:00
fi