Add 1.10.11 Changelog & Update UPDATING.md (#9757)

2020-07-11 19:15:02 +01:00 · 2020-07-11 19:15:02 +01:00 · 619ab69657
--- a/.pre-commit-config.yaml
+++ b/.pre-commit-config.yaml
@ -269,7 +269,8 @@ repos:
          ^airflow/providers/apache/hive/operators/hive_stats.py$|
          ^airflow/providers/apache/hive/PROVIDERS_CHANGES_*|
          ^airflow/providers/apache/hive/README.md$|
-          ^tests/providers/apache/cassandra/hooks/test_cassandra.py
+          ^tests/providers/apache/cassandra/hooks/test_cassandra.py$|
+          ^CHANGELOG.txt$
      - id: consistent-pylint
        language: pygrep
        name: Check for inconsitent pylint disable/enable without space
--- a/CHANGELOG.txt
+++ b/CHANGELOG.txt
@ -1,3 +1,198 @@
+Airflow 1.10.11, 2020-07-10
+-----------------------------
+
+New Features
+""""""""""""
+
+- Add task instance mutation hook (#8852)
+- Allow changing Task States Colors (#9520)
+- Add support for AWS Secrets Manager as Secrets Backend (#8186)
+- Add airflow info command to the CLI (#8704)
+- Add Local Filesystem Secret Backend (#8596)
+- Add Airflow config CLI command (#8694)
+- Add Support for Python 3.8 (#8836)(#8823)
+- Allow K8S worker pod to be configured from JSON/YAML file (#6230)
+- Add quarterly to crontab presets (#6873)
+- Add support for ephemeral storage on KubernetesPodOperator (#6337)
+- Add AirflowFailException to fail without any retry (#7133)
+- Add SQL Branch Operator (#8942)
+
+Bug Fixes
+"""""""""
+
+- Use NULL as dag.description default value (#7593)
+- BugFix: DAG trigger via UI error in RBAC UI (#8411)
+- Fix logging issue when running tasks (#9363)
+- Fix JSON encoding error in DockerOperator (#8287)
+- Fix alembic crash due to typing import (#6547)
+- Correctly restore upstream_task_ids when deserializing Operators (#8775)
+- Correctly store non-default Nones in serialized tasks/dags (#8772)
+- Correctly deserialize dagrun_timeout field on DAGs (#8735)
+- Fix tree view if config contains " (#9250)
+- Fix Dag Run UI execution date with timezone cannot be saved issue (#8902)
+- Fix Migration for MSSQL (#8385)
+- RBAC ui: Fix missing Y-axis labels with units in plots (#8252)
+- RBAC ui: Fix missing task runs being rendered as circles instead (#8253)
+- Fix: DagRuns page renders the state column with artifacts in old UI (#9612)
+- Fix task and dag stats on home page (#8865)
+- Fix the trigger_dag api in the case of nested subdags (#8081)
+- UX Fix: Prevent undesired text selection with DAG title selection in Chrome (#8912)
+- Fix connection add/edit for spark (#8685)
+- Fix retries causing constraint violation on MySQL with DAG Serialization (#9336)
+- [AIRFLOW-4472] Use json.dumps/loads for templating lineage data (#5253)
+- Restrict google-cloud-texttospeach to <v2 (#9137)
+- Fix pickling failure when spawning processes (#8671)
+- Pin Version of Azure Cosmos to <4 (#8956)
+- Azure storage 0.37.0 is not installable any more (#8833)
+- Fix modal_backdrop z-index in the UI (#7313)
+- Fix Extra Links in Gannt View (#8308)
+- Bug fix for EmrAddStepOperator init with cluster_name error (#9235)
+- Fix KubernetesPodOperator pod name length validation (#8829)
+- Fix non updating DAG code by checking against last modification time (#8266)
+- BugFix: Unpausing a DAG with catchup=False creates an extra DAG run (#8776)
+
+
+Improvements
+""""""""""""
+
+- Improve add_dag_code_table migration (#8176)
+- Persistent display/filtering of DAG status (#8106)
+- Set unique logger names (#7330)
+- Update the version of cattrs from 0.9 to 1.0 to support Python 3.8 (#7100)
+- Reduce response payload size of /dag_stats and /task_stats (#8655)
+- Add authenticator parameter to snowflake_hook (#8642)
+- Show "Task Reschedule" table in Airflow Webserver (#9521)
+- Change worker_refresh_interval fallback to default of 30 (#9588)
+- Use pfromat instead of str to render arguments in WebUI (#9587)
+- Simplify DagFileProcessorManager (#7521)
+- Reload gunicorn when plugins has beeen changed (#8997)
+- Fix the default value for store_dag_code (#9554)
+- Add support for fetching logs from running pods (#8626)
+- Persist start/end date and duration for DummyOperator Task Instance (#8663)
+- Ensure "started"/"ended" in tooltips are not shown if job not started (#8667)
+- Add context to execution_date_fn in ExternalTaskSensor (#8702)
+- Avoid color info in response of ``/dag_stats`` & ``/task_stats`` (#8742)
+- Make loading plugins from entrypoint fault-tolerant (#8732)
+- Refactor Kubernetes worker config (#7114)
+- Add default ``conf`` parameter to Spark JDBC Hook (#8787)
+- Allow passing backend_kwargs to AWS SSM client (#8802)
+- Filter dags by clicking on tag (#8897)
+- Support k8s auth method in Vault Secrets provider (#8640)
+- Monitor pods by labels instead of names (#6377)
+- Optimize count query on /home (#8729)
+- Fix json string escape in tree view (#8551)
+- Add TaskInstance state to TI Tooltip to be colour-blind friendlier (#8910)
+- Add a tip to trigger DAG screen (#9049)
+- Use Markup for htmlcontent for landing_times (#9242)
+- Pinning max pandas version to 2.0 (lesser than) to allow pandas 1.0 (#7954)
+- Update example webserver_config.py to show correct CSRF config (#8944)
+- Fix displaying Executor Class Name in "Base Job" table (#8679)
+- Use existing DagBag for 'dag_details' & 'trigger' Endpoints (#8501)
+- Flush pending Sentry exceptions before exiting (#7232)
+- Display DAG run conf in the list view (#6794)
+- Fix performance degradation when updating dagrun state (#8435)
+- Don't use the ``|safe`` filter in code, it's risky (#9180)
+- Validate only task commands are run by executors (#9178)
+- Show Deprecation warning on duplicate Task ids (#8728)
+- Move DAG._schedule_interval logic out of ``DAG.__init__`` (#8225)
+- Make retrieving Paused Dag ids a separate method (#7587)
+- Bulk fetch paused_dag_ids (#7476)
+- Add a configurable DAGs volume mount path for Kubernetes (#8147)
+- Add schedulername option for KubernetesPodOperator (#6088)
+- Support running git sync container as root (#6312)
+- Add extra options for Slack Webhook operator and Slack hook (#9409)
+- Monkey patch greenlet celery pools (#8559)
+- Decrypt secrets from SystemsManagerParameterStoreBackend (#9214)
+- Prevent clickable sorting on non sortable columns in TI view (#8681)
+- Make hive macros py3 compatible (#8598)
+- Fix SVG tooltip positioning with custom scripting (#8269)
+- Avoid unnecessary sleep to maintain local task job heart rate (#6553)
+- Include some missing RBAC roles on User and Viewer roles (#9133)
+- Show Dag's Markdown docs on Tree View (#9448)
+- Improved compatibility with Python 3.5+ - Convert signal.SIGTERM to int (#9207)
+- Add 'main' param to template_fields in DataprocSubmitPySparkJobOperator (#9154)
+- Make it possible to silence warnings from Airflow (#9208)
+- Remove redundant count query in BaseOperator.clear() (#9362)
+- Fix DB migration message (#8988)
+- Fix awkward log info in dbapi_hook (#8482)
+- Fix Celery default to no longer allow pickle (#7205)
+- Further validation that only task commands are run by executors (#9240)
+- Remove vendored nvd3 and slugify libraries (#9136)
+- Enable configurable git sync depth  (#9094)
+- Reduce the required resources for the Kubernetes's sidecar (#6062)
+- Refactor K8S codebase with k8s API models (#5481)
+- Move k8sexecutor out of contrib to closer match master (#8904)
+- Allow filtering using "event" and "owner" in "Log" view (#4881)
+- Add Yandex.Cloud custom connection to 1.10 (#8791)
+- Add table-hover css class to DAGs table (#5033)
+- Show un/pause errors in dags view. (#7669)
+- Restructure database queries on /home (#4872)
+- Add Cross Site Scripting defence (#6913)
+- Make Gantt tooltip the same as Tree and Graph view (#8220)
+- Add config to only delete worker pod on task failure (#7507)(#8312)
+- Remove duplicate error message on chart connection failure (#8476)
+- Remove default value spark_binary (#8508)
+- Expose Airflow Webserver Port in Production Docker Image (#8228)
+- Commit after each alembic migration (#4797)
+- KubernetesPodOperator fixes and test (#6524)
+- Docker Image: Add ADDITIONAL_AIRFLOW_EXTRAS (#9032)
+- Docker Image: Add ADDITIONAL_PYTHON_DEPS (#9031)
+- Remove httplib2 from Google requirements (#9194)
+- Merging multiple sql operators (#9124)
+- Adds hive as extra in pyhive dependency (#9075)
+- Change default auth for experimental backend to deny_all (#9611)
+- Restrict changing XCom values from the Webserver (#9614)
+- Add __repr__ for DagTag so tags display properly in /dagmodel/show (#8719)
+- Functionality to shuffle HMS connections used by HiveMetastoreHook facilitating load balancing (#9280)
+- Expose SQLAlchemy's connect_args and make it configurable (#6478)
+
+Doc only changes
+""""""""""""""""
+
+- Add docs on using DAGRun.conf (#9578)
+- Enforce code-block directives in doc (#9443)
+- Carefully parse warning messages when building documentation (#8693)
+- Make KubernetesPodOperator clear in docs (#8444)
+- Improve language in Pod Mutation Hook docs (#8445)
+- Fix formatting of Pool docs in concepts.rst (#8443)
+- Make doc clearer about Airflow Variables using Environment Variables (#8427)
+- Fix pools doc for LocalExecutor (#7643)
+- Add Local and Sequential Executors to Doc (#8084)
+- Add documentation for CLI command airflow dags test (#8251)
+- Fix typo in DAG Serialization documentation (#8317)
+- Add scheduler in production section (#7351)
+- Add a structural dag validation example (#6727)
+- Adding Task re-run documentation (#6295)
+- Fix outdated doc on settings.policy (#7532)
+- Add docs about reload_on_plugin_change opiton (#9575)
+- Add copy button to Code Blocks in Airflow Docs (#9450)
+- Update commands in docs for v1.10+ (#9585)
+- Add more info on dry-run CLI option (#9582)
+- Document default timeout value for SSHOperator (#8744)
+- Fix docs on creating CustomOperator (#8678)
+- Enhanced documentation around Cluster Policy (#8661)
+- Use sphinx syntax in concepts.rst (#7729)
+- Update README to remove Py 3.8 limitation for Master (#9451)
+- Add note about using dag_run.conf in BashOperator (#9143)
+- Improve tutorial - Include all imports statements (#8670)
+- Added more precise Python requirements to README.md (#8455)
+- Fix Airflow Stable version in README.md (#9360)
+- Update AWS connection example to show how to set from env var (#9191)
+- Fix list formatting of plugins doc. (#8873)
+- Add 'Version Added' on Secrets Backend docs (#8264)
+- Simplify language re roll-your-own secrets backend (#8257)
+- Add installation description for repeatable PyPi installation (#8513)
+- Add note extra links only render on when using RBAC webserver  (#8788)
+- Remove unused airflow import from docs (#9274)
+- Add PR/issue note in Contribution Workflow Example (#9177)
+- Don't use the term "whitelist" - language matters (#9174)
+- Add docs to change Colors on the Webserver (#9607)
+- Change 'initiate' to 'initialize' in installation.rst (#9619)
+- Replace old Variables View Screenshot with new (#9620)
+- Replace old SubDag zoom screenshot with new (#9621)
+- Update docs about the change to default auth for experimental API (#9617)
+
+
 Airflow 1.10.10, 2020-04-09
 ---------------------------

--- a/UPDATING.md
+++ b/UPDATING.md
@ -26,6 +26,7 @@ assists users migrating to a new version.
 **Table of contents**

 - [Airflow Master](#airflow-master)
+- [Airflow 1.10.11](#airflow-11011)
 - [Airflow 1.10.10](#airflow-11010)
 - [Airflow 1.10.9](#airflow-1109)
 - [Airflow 1.10.8](#airflow-1108)
@ -317,11 +318,6 @@ plugins =
  airflow.mypy.plugin.decorators
 ```

-### XCom Values can no longer be added or changed from the Webserver
-
-Since XCom values can contain pickled data, we would no longer allow adding or
-changing XCom values from the UI.
-
 ### Use project_id argument consistently across GCP hooks and operators

 - Changed order of arguments in DataflowHook.start_python_dataflow. Uses
@ -398,10 +394,6 @@ Remove unnecessary parameter ``open`` in PostgresHook function copy_expert for p

 Change parameter name from ``visibleTo`` to ``visible_to`` in OpsgenieAlertOperator for pylint compatible

-### Use NULL as default value for dag.description
-
-Now use NULL as default value for dag.description in dag table
-
 ### Assigning task to a DAG using bitwise shift (bit-shift) operators are no longer supported

 Previously, you could assign a task to a DAG as follows:
@ -1450,6 +1442,19 @@ Now the `dag_id` will not appear repeated in the payload, and the response forma
 }
 ```

+## Airflow 1.10.11
+
+### Use NULL as default value for dag.description
+
+Now use NULL as default value for dag.description in dag table
+
+### Restrict editing DagRun State in the old UI (Flask-admin based UI)
+
+Before 1.10.11 it was possible to edit DagRun State in the `/admin/dagrun/` page
+ to any text.
+
+In Airflow 1.10.11+, the user can only choose the states from the list.
+
 ### Experimental API will deny all request by default.

 The previous default setting was to allow all API requests without authentication, but this poses security
@ -1466,6 +1471,11 @@ the previous behaviour on a new install by setting this in your airflow.cfg:
 auth_backend = airflow.api.auth.backend.default
 ```

+### XCom Values can no longer be added or changed from the Webserver
+
+Since XCom values can contain pickled data, we would no longer allow adding or
+changing XCom values from the UI.
+
 ## Airflow 1.10.10

 ### Setting Empty string to a Airflow Variable will return an empty string
--- a/airflow/config_templates/config.yml
+++ b/airflow/config_templates/config.yml
@ -149,7 +149,7 @@
        This is useful when you want to configure db engine args that SqlAlchemy won't parse
        in connection string.
        See https://docs.sqlalchemy.org/en/13/core/engines.html#sqlalchemy.create_engine.params.connect_args
-      version_added: ~
+      version_added: 1.10.11
      type: string
      example: ~
      default: ~
@ -787,7 +787,7 @@
      description: |
        If set to True, Airflow will track files in plugins_folder directory. When it detects changes,
        then reload the gunicorn.
-      version_added: ~
+      version_added: 1.10.11
      type: boolean
      example: ~
      default: "False"
@ -1797,7 +1797,7 @@
    - name: pod_template_file
      description: |
        Path to the YAML pod file. If set, all other kubernetes-related fields are ignored.
-      version_added: ~
+      version_added: 1.10.11
      type: string
      example: ~
      default: ""
@ -1826,7 +1826,7 @@
      description: |
        If False (and delete_worker_pods is True),
        failed worker pods will not be deleted so users can investigate them.
-      version_added: ~
+      version_added: 1.10.11
      type: string
      example: ~
      default: "False"
@ -1920,7 +1920,7 @@
    - name: dags_volume_mount_point
      description: |
        For either git sync or volume mounted DAGs, the worker will mount the volume in this path
-      version_added: ~
+      version_added: 1.10.11
      type: string
      example: ~
      default: ""