a50e36ddc8
Format Jinja blocks like SQL blocks. ( #3952 )
2023-06-15 23:42:28 +00:00
f02ab7564d
Add `stripe_subscriptions_changelog_v1` ETL (DENG-974) ( #3950 )
...
* Add `stripe_external` ETLs for `subscription_discount`, `subscription_tax_rate`, and `tax_rate`.
* Add `stripe_subscriptions_changelog_v1` ETL.
* Regenerate `bqetl_subplat` DAG.
* Quote `timestamp` column name.
2023-06-15 22:12:30 +00:00
953529a3a5
Process subsequent checks for a table if a prior check fails for that table ( #3943 )
...
* Process subsequent checks for a table if a prior check fails for that table.
* Updated to use sqlparse to parse checks.sql.
2023-06-15 21:12:08 +00:00
25a20bdfbf
Regex for matching UDF names ( #3949 )
2023-06-15 18:40:16 +00:00
faf4dc8269
Dryrun date param fixes ( #3942 )
...
* Always rewrite dryrun date query params as `submission_date`.
* Quote date partition column in dryrun to get schema.
2023-06-14 21:31:18 +00:00
d82acc1856
Simplify GLAM template for getting the latest version ( #3933 )
...
* Simplify GLAM template for getting the latest version
* Add comment about using buildhub2 data for Fenix
2023-06-14 20:03:01 +00:00
166e0d873f
Bump sqlglot from 14.1.1 to 16.1.0 ( #3937 )
...
Bumps [sqlglot](https://github.com/tobymao/sqlglot ) from 14.1.1 to 16.1.0.
- [Changelog](https://github.com/tobymao/sqlglot/blob/main/CHANGELOG.md )
- [Commits](https://github.com/tobymao/sqlglot/compare/v14.1.1...v16.1.0 )
---
updated-dependencies:
- dependency-name: sqlglot
dependency-type: direct:production
update-type: version-update:semver-major
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-06-14 18:07:34 +00:00
d9c68a48d1
glam: Partition clients_histogram_aggregates by sample_id ( #3868 )
...
* glam: Partition clients_histogram_aggregates by sample_id (has been running like this since April 3 from a different branch)
* glam: add description and eol to init
* glam: Partition clients_histogram_aggregates by sample_id (has been running like this since April 3 from a different branch)
* glam: add description and eol to init
* add init.sql to missing tbls
* Add schema.yaml
* increase ci output timeout to 30m
* remove init.sql to prevent ci from trying to derive schema from it and break
* Fix schema.yaml files
* Revert output timeout to default
2023-06-14 16:37:59 +00:00
57665fdd2f
Bump google-cloud-bigquery from 3.11.0 to 3.11.1 ( #3940 )
...
Bumps [google-cloud-bigquery](https://github.com/googleapis/python-bigquery ) from 3.11.0 to 3.11.1.
- [Release notes](https://github.com/googleapis/python-bigquery/releases )
- [Changelog](https://github.com/googleapis/python-bigquery/blob/main/CHANGELOG.md )
- [Commits](https://github.com/googleapis/python-bigquery/compare/v3.11.0...v3.11.1 )
---
updated-dependencies:
- dependency-name: google-cloud-bigquery
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-06-14 16:19:51 +00:00
a55c8eb112
Bump pre-commit from 3.3.2 to 3.3.3 ( #3938 )
...
Bumps [pre-commit](https://github.com/pre-commit/pre-commit ) from 3.3.2 to 3.3.3.
- [Release notes](https://github.com/pre-commit/pre-commit/releases )
- [Changelog](https://github.com/pre-commit/pre-commit/blob/main/CHANGELOG.md )
- [Commits](https://github.com/pre-commit/pre-commit/compare/v3.3.2...v3.3.3 )
---
updated-dependencies:
- dependency-name: pre-commit
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-06-14 16:19:31 +00:00
c69fee0b5f
DENG-941 initial impl of check rendering and execution. ( #3885 )
...
* initial impl
* Updated based on PR feedback
* Moved check from query to separate command
* Expanded from --partition option to generic --parameter option
* Removed `query check` command (check moved to new command)
* Update bigquery_etl/cli/check.py
remove date param format check
Co-authored-by: Anna Scholtz <anna@scholtzan.net>
* Removed 'parameter' parameter, everything is passed through ctx.args and then converted to a dict for Jinja rendering. There are no restrictions on ctx.args values.
* Merge error
---------
Co-authored-by: Anna Scholtz <anna@scholtzan.net>
2023-06-13 19:31:59 +00:00
65365226b5
Don't deduplicate query arguments ( #3935 )
2023-06-13 17:34:00 +00:00
2f8e619c16
Bump gcsfs from 2023.5.0 to 2023.6.0 ( #3934 )
...
Bumps [gcsfs](https://github.com/fsspec/gcsfs ) from 2023.5.0 to 2023.6.0.
- [Commits](https://github.com/fsspec/gcsfs/compare/2023.5.0...2023.6.0 )
---
updated-dependencies:
- dependency-name: gcsfs
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-06-13 17:13:58 +00:00
b2a06b8779
if --parameter passed we set use_legacy_sql option to False by default and bq called with query by default if not explicitly passed in the bqetl query run command ( #3922 )
2023-06-13 08:53:03 +00:00
d68cace7b0
DENG-984 Added backfill staging dataset ( #3904 )
...
* Added backfill staging dataset
* Updated dataset metadata
* Update sql/moz-fx-data-shared-prod/backfills_staging_derived/dataset_metadata.yaml
Co-authored-by: Alexander <anicholson@mozilla.com>
---------
Co-authored-by: Alexander <anicholson@mozilla.com>
2023-06-12 21:08:49 +00:00
f6954bea1c
Bump types-ujson from 5.7.0.5 to 5.8.0.0 ( #3928 )
...
Bumps [types-ujson](https://github.com/python/typeshed ) from 5.7.0.5 to 5.8.0.0.
- [Commits](https://github.com/python/typeshed/commits )
---
updated-dependencies:
- dependency-name: types-ujson
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-06-12 17:33:47 +00:00
9c9cacbe4e
Bump pytest from 7.3.1 to 7.3.2 ( #3929 )
...
Bumps [pytest](https://github.com/pytest-dev/pytest ) from 7.3.1 to 7.3.2.
- [Release notes](https://github.com/pytest-dev/pytest/releases )
- [Changelog](https://github.com/pytest-dev/pytest/blob/main/CHANGELOG.rst )
- [Commits](https://github.com/pytest-dev/pytest/compare/7.3.1...7.3.2 )
---
updated-dependencies:
- dependency-name: pytest
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-06-12 17:33:44 +00:00
442cb132e9
Bump ujson from 5.7.0 to 5.8.0 ( #3931 )
...
Bumps [ujson](https://github.com/ultrajson/ultrajson ) from 5.7.0 to 5.8.0.
- [Release notes](https://github.com/ultrajson/ultrajson/releases )
- [Commits](https://github.com/ultrajson/ultrajson/compare/5.7.0...5.8.0 )
---
updated-dependencies:
- dependency-name: ujson
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-06-12 17:33:41 +00:00
b9d01ca959
DENG-990 Refractor backfill cli commands ( #3924 )
...
* Refractored backfill cli commands
* Adjusted validate command
2023-06-12 17:24:35 +00:00
02afdfb443
Ignore comments when detecting dependency table names. ( #3927 )
...
Otherwise the names of unaliased table references followed by a comment will incorrectly include the comment as part of the name.
2023-06-10 20:20:02 +00:00
58c96b4246
DENG-815 Add backfill info cli command ( #3915 )
...
* Added backfill info command
* Update bigquery_etl/cli/backfill.py
Co-authored-by: Alexander <anicholson@mozilla.com>
* Fixed status click choice
* Added backfill str method
* Added new backfill utils files
* Update bigquery_etl/cli/backfill.py
Co-authored-by: Alexander <anicholson@mozilla.com>
* Update bigquery_etl/cli/backfill.py
Co-authored-by: Alexander <anicholson@mozilla.com>
* Removed status default
---------
Co-authored-by: Alexander <anicholson@mozilla.com>
2023-06-09 17:32:32 +00:00
b7b7c23913
Preserve the order of column schema properties in `schema.yaml` files. ( #3923 )
...
When using `bqetl query schema update` to create a new `schema.yaml` file, BigQuery returns the column schema properties in a sensible order (`name`, `type`, `mode`, `fields`), but our `schema.yaml` output has been sorting those properties alphabetically which makes it much less readable.
Also, when using `bqetl query schema update` to update an existing `schema.yaml` file, this will now preserve whatever order the column schema properties were in.
2023-06-09 16:15:40 +00:00
8423c7ad2e
Use baseline_clients_daily instead of ping and first_seen for fenix_android_clients ( #3910 )
...
* Change source for first_seen and baseline to baseline_clients_daily
* Edit tests and schemas
* Update to fenix.baseline_clients_daily
2023-06-08 14:50:14 +00:00
eb02488f34
Fix google sheets metadata and change from "google_sheet" to "google_sheets" for consistency with google ( #3914 )
2023-06-07 19:25:31 +00:00
d8f1dbf1ff
Bump mozilla-metric-config-parser from 2023.6.1 to 2023.6.3 ( #3911 )
...
Bumps [mozilla-metric-config-parser](https://github.com/mozilla/metric-config-parser ) from 2023.6.1 to 2023.6.3.
- [Release notes](https://github.com/mozilla/metric-config-parser/releases )
- [Commits](https://github.com/mozilla/metric-config-parser/compare/2023.6.1...2023.6.3 )
---
updated-dependencies:
- dependency-name: mozilla-metric-config-parser
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-06-07 16:23:04 +00:00
8e3b20605e
including activation info for all users not just a specific partition and partition_field no longer required for new_profile_activation_v2 ( #3912 )
2023-06-07 12:33:28 +00:00
e7eaa78309
feat(DENG-997): apple_ads spend per activation ( #3878 )
...
* added a view ad_spend_per_activation view which contains stats about ad campaign performance
* updated ad performance query and renamed the view
* implemented fbertsch feedback into the query
* fixed DDL for the ad_activation_performance view
2023-06-07 09:22:11 +00:00
d6a55664d0
Revert "Simplify GLAM template for getting latest versions ( #3880 )" ( #3908 )
...
This reverts commit 8ad45a0592
2023-06-06 18:41:03 +00:00
e2cc3d1d31
Bump mozilla-metric-config-parser from 2023.5.1 to 2023.6.1 ( #3905 )
...
Bumps [mozilla-metric-config-parser](https://github.com/mozilla/metric-config-parser ) from 2023.5.1 to 2023.6.1.
- [Release notes](https://github.com/mozilla/metric-config-parser/releases )
- [Commits](https://github.com/mozilla/metric-config-parser/compare/2023.5.1...2023.6.1 )
---
updated-dependencies:
- dependency-name: mozilla-metric-config-parser
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-06-06 15:38:12 +00:00
b0e01a27e7
Bump cattrs from 23.1.1 to 23.1.2 ( #3886 )
...
Bumps [cattrs](https://github.com/python-attrs/cattrs ) from 23.1.1 to 23.1.2.
- [Changelog](https://github.com/python-attrs/cattrs/blob/main/HISTORY.md )
- [Commits](https://github.com/python-attrs/cattrs/compare/v23.1.1...v23.1.2 )
---
updated-dependencies:
- dependency-name: cattrs
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-06-06 15:37:51 +00:00
f333de253b
Bump google-cloud-bigquery from 3.10.0 to 3.11.0 ( #3887 )
...
Bumps [google-cloud-bigquery](https://github.com/googleapis/python-bigquery ) from 3.10.0 to 3.11.0.
- [Release notes](https://github.com/googleapis/python-bigquery/releases )
- [Changelog](https://github.com/googleapis/python-bigquery/blob/main/CHANGELOG.md )
- [Commits](https://github.com/googleapis/python-bigquery/compare/v3.10.0...v3.11.0 )
---
updated-dependencies:
- dependency-name: google-cloud-bigquery
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-06-06 15:37:39 +00:00
2f3e20512b
Bump cryptography from 39.0.1 to 41.0.0 ( #3896 )
...
Bumps [cryptography](https://github.com/pyca/cryptography ) from 39.0.1 to 41.0.0.
- [Changelog](https://github.com/pyca/cryptography/blob/main/CHANGELOG.rst )
- [Commits](https://github.com/pyca/cryptography/compare/39.0.1...41.0.0 )
---
updated-dependencies:
- dependency-name: cryptography
dependency-type: indirect
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-06-06 15:37:24 +00:00
71e7201e65
feat(): added support for `--log-level` to bqetl query command and using logging instead of print() ( #3891 )
...
* added support for --log-level to bqetl query command and updated print statements to be log statements
* now --log-level flag is a bqetl global flag
* fixing linter errors
* Update bigquery_etl/cli/__init__.py
Co-authored-by: Anna Scholtz <anna@scholtzan.net>
* Update bigquery_etl/cli/__init__.py
Co-authored-by: Anna Scholtz <anna@scholtzan.net>
* fixed indentation of --log-level option
---------
Co-authored-by: Anna Scholtz <anna@scholtzan.net>
2023-06-06 09:16:41 +00:00
1d3030e698
Remove ZetaSQL kludges. ( #3898 )
...
ZetaSQL was removed in #3755 .
2023-06-05 18:03:07 +00:00
505c895f62
GROWTH-41 Add last_seen columns to firefox_android_clients ( #3863 )
...
* Added last_reported columns
* Fixed tests
* Added missing locale field
2023-06-05 16:37:59 +00:00
1a527d743e
Fix `bqetl stage` table ID quoting ( #3899 )
...
* Fix `bqetl stage` ID quoting.
Quoting the entire table ID breaks cases where an unaliased table name is used to qualify a column reference.
* Have `bqetl stage` preserve fully quoted references.
* Simplify regular expressions for fully quoted references.
* Compile all reference replacement regular expressions for performance.
2023-06-05 16:24:14 +00:00
5dc9e405a8
Deng 1000 docs how to change control ( #3903 )
...
* DENG-1000. Update docs with change control guide.
* DENG-1000. Update docs with change control guide.
* DENG-1000. Add sample PR to docs.
* DENG-1000. Add sample PR to docs.
* DENG-1000. Fix syntax of MD file.
---------
Co-authored-by: Lucia Vargas <lvargas@mozilla.com>
2023-06-05 15:53:16 +00:00
830e887de9
Remove NULL suffix from campaigns ( #3902 )
2023-06-05 15:52:42 +00:00
dfef07ee40
DENG-1000. Update docs with change control guide. ( #3901 )
...
* DENG-1000. Update docs with change control guide.
* DENG-1000. Update docs with change control guide.
* DENG-1000. Add sample PR to docs.
---------
Co-authored-by: Lucia Vargas <lvargas@mozilla.com>
2023-06-05 14:53:20 +00:00
8ad45a0592
Simplify GLAM template for getting latest versions ( #3880 )
...
* Simplify GLAM latest version template
* Use buildhub2 table instead
---------
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-06-05 14:23:21 +00:00
5330dd19da
Move schema and initialization logic for firefox_android_clients_v1 to metadata ( #3893 )
...
* Move schema and initializing logic for firefox_android_clients_v1 to metadata
* bqetl query schema update
* Stage table on init.sql change as well
2023-06-05 13:14:03 +00:00
cfca76f9c4
Sync currency exchange rates to support VPN wave 6 (DS-2907) ( #3897 )
...
* Sync currency exchange rates to support VPN wave 6 (DS-2907).
* Regenerate `bqetl_subplat` DAG.
2023-06-02 21:38:28 +00:00
c70a17144a
Save current SubPlat ETL views logic in versioned ETLs (DENG-973) ( #3883 )
...
* Save current SubPlat ETL views logic in versioned ETLs (DENG-973).
* Add `incremental` labels to the new tables.
* List all CJMS ETLs to dryrun-skip rather than using `glob`.
The `glob` approach doesn't currently work well with the CI staging process.
2023-06-02 19:53:49 +00:00
3f787df986
Fix destination paths when copying stage files in CI ( #3895 )
2023-06-02 19:06:01 +00:00
2c36487712
fixed formatting for some sql files ( #3894 )
2023-06-02 17:16:26 +00:00
dd4789c8aa
DENG-970 Only Glean in Focus Android view. ( #3877 )
...
* DENG-970 Only Glean in Focus Android view.
* DENG-970 Only Glean in Focus Android view.
* DENG-970 Only Glean in Focus Android view.
* DENG-970 Only Glean in Focus Android view.
* DENG-970 Only Glean in Focus Android view.
* DENG-970 Only Glean in Focus Android view.
* DENG-970 CI fix
* DENG-970 CI failure fix. Related to issue 3889.
* Fix UDF dependencies deploy on stage
* DENG-970 Revert specific calling to dataset for UDF.
---------
Co-authored-by: Lucia Vargas <lvargas@mozilla.com>
Co-authored-by: Brad Ochocki <brad.ochocki@gmail.com>
Co-authored-by: Anna Scholtz <anna@scholtzan.net>
2023-06-02 16:37:40 +00:00
f52700dcfe
Format transaction statements properly ( #3892 )
...
* Format transaction statements properly.
* Test transaction statement formatting.
2023-06-02 16:36:11 +00:00
19bcffa8f7
During stage don't rename test dependencies that have already been renamed ( #3890 )
2023-06-02 15:13:38 +00:00
1ea0b037e6
fxa_users_services_first_seen view now using v2 version of the table which is actively updated ( #3888 )
2023-06-02 11:53:40 +00:00
561935a6cc
Revert "feat(DENG-980): fxa_users_services_first_seen view updated to use v2 ( #3870 )" ( #3884 )
...
This reverts commit 058ab9d98a
2023-06-01 20:02:40 +00:00
Sean Rose
Glenda Leonard
Anna Scholtz
Linh Nguyen
dependabot[bot]
Eduardo Filho
Daniel Thorn
kik-kik
Winnie Chan
Alexander
Curtis Morales
Lucia
Frank Bertsch