* Move `stripe_subscriptions_changelog_v1` ETL to `stripe_external` dataset.
* Include all subscription metadata in `stripe_external.subscriptions_changelog_v1`.
* Bug 1741487 - Rename url2 and related fields in stable views
This removes the following unpopulated fields from Glean views: `metrics.url`, `metrics.text`, `metrics.jwe`, and `metrics.labeled_rate`. If any of these metrics exist in the source table under `2`-suffixed name, it is also aliased to its original name (`url2` to `url` and so on).
Suffixed fields are still preserved until view consumers migrate.
* Remove redundant comma from generated sql
* Ignore missing fields in views if any of them were removed
* added a todo comment
* Added additional context around why we are excluding some of the non-suffixed fields and why alising to remove suffix 2 from some fields
---------
Co-authored-by: Arkadiusz Komarzewski <akomarzewski@mozilla.com>
Co-authored-by: Anna Scholtz <anna@scholtzan.net>
* take rbaffourawuah@mozilla.com off of email list for DAG
* remove rbaffourawuah@mozilla.com from dags.yaml
* fix formatting and remove another instance of rbaffourawuah@mozilla.com from DAG
* fix order of emails
* modify name to be more explicity
* rename view and folder
* update view name
* update view location
* update view name
* add metadata, schema yamls and query.py
* created adjust_derived namespace
* add query.py, metadata, schema, dataset for testing
* delete extraneous file, update DAG name
* modify bqetl_adjust DAG redux
* update DAG name, take out '_derived'
* update table name in view
* standardize table names across files
* regenerate DAG
* update schema in both locations
* add query.py, metadata, schema yaml files
* take put extraneous print statements, update datasets to be 'adjust' or 'adjust_derived'
* add submission date to date_partition_parameter
* update table name to be just one table
* add DAG for adjust_derived
* add bq_etl adjust_derived DAG to yaml file
* add note about API token
* revert changes to bqetl.adjust.py
* use proper tast_id
* fix start dates
* add python command and docker image
* add python command and docker image
* delete extraneous code
* comment out docker part in old adjust dag
* add whitespace, delete extraneous code
* Update sql/moz-fx-data-shared-prod/adjust/adjust_derived/view.sql
Co-authored-by: Lucia <30448600+lucia-vargas-a@users.noreply.github.com>
* Update sql/moz-fx-data-shared-prod/adjust_derived/adjust_derived_v1/query.py
Co-authored-by: Lucia <30448600+lucia-vargas-a@users.noreply.github.com>
* Update sql/moz-fx-data-shared-prod/adjust_derived/adjust_derived_v1/query.py
Co-authored-by: kik-kik <42538694+kik-kik@users.noreply.github.com>
* updated logic to check if response dictionary is not empty, moved view out of nested folder, added token ownership statement to metadata file, turned off email retry in dags.yaml, separated out clean up of json to its own function
* take out extraneous if statement and move else statement
* reorder where comment is to make more sense
* more description as to why we're using mhirose's API token
* take out periods
* Update sql/moz-fx-data-shared-prod/adjust_derived/adjust_derived_v1/metadata.yaml
Co-authored-by: Anna Scholtz <anna@scholtzan.net>
* combine adjust DAGs
* change logic for query_export check loop continuance, adapt metadata.yamls
* add blank parameters test
* Update sql/moz-fx-data-shared-prod/adjust_derived/adjust_derived_v1/metadata.yaml
Co-authored-by: Anna Scholtz <anna@scholtzan.net>
* add arguments to metadata.yaml
* remove external table reference
* refactor to add date parameter
* refactor based on Circle CI's advice
* Update sql/moz-fx-data-shared-prod/adjust_derived/adjust_derived_v1/query.py
Co-authored-by: kik-kik <42538694+kik-kik@users.noreply.github.com>
* Update sql/moz-fx-data-shared-prod/adjust_derived/adjust_derived_v1/query.py
Co-authored-by: kik-kik <42538694+kik-kik@users.noreply.github.com>
* take out TODO comment
---------
Co-authored-by: kik-kik <kignasiak@mozilla.com>
Co-authored-by: Lucia <30448600+lucia-vargas-a@users.noreply.github.com>
Co-authored-by: kik-kik <42538694+kik-kik@users.noreply.github.com>
Co-authored-by: Anna Scholtz <anna@scholtzan.net>
* Add `synced_at` column to `stripe_subscriptions_changelog_v1`.
* Tweak `stripe_subscriptions_changelog_v1` tax rate and discount joins to only include those that existed when the change happened.
* Parse subscription metadata in `stripe_subscriptions_changelog_v1`.
* Add `stripe_external.invoice_line_item_v1` ETL.
* Add `stripe_subscriptions_revised_changelog_v1` ETL.
* Add `stripe_subscriptions_history_v2` ETL.
Anna Scholtz
dependabot[bot]
Marlene Hirose
akkomar
Lucia
Sean Rose
kik-kik
Leli
Glenda Leonard
Alexander
Daniel Thorn
Winnie Chan
Frank Bertsch