9e794dee20
update_ios_feature_usage_events_w_ping_parse_time_logic ( #4968 )
...
* update_ios_feature_usage_events_w_ping_parse_time_logic
* Update query.sql
* Update schema.yaml
* Update query.sql
* Update query.sql
2024-02-09 14:56:00 +01:00
9049398c36
adding firefox_ios_derived.retention_clients to shredder config ( #4993 )
2024-02-09 12:14:00 +01:00
133adfd472
Don't overwrite existing `gcp.json` files unless that's actually necessary ( #4997 )
2024-02-08 15:37:28 -08:00
eeb78dc7e4
DENG-2492 update the product download types and the clustering ( #4996 )
2024-02-08 15:47:03 -06:00
c05ac1d661
Stop using data-eng-circleci-tests context in CI ( #4884 )
2024-02-08 21:35:55 +00:00
638e4e2065
DENG-2492 - fix time_on_site formula to convert to seconds ( #4994 )
...
* DENG-2492 - fix time_on_site formula to convert to seconds
* DENG-2492 update SQL format with new formula change
2024-02-08 14:51:04 -06:00
e741b7a967
Add clustering and partitioning to meta_attribution_country_counts_v1 init.sql ( #4995 )
2024-02-08 12:50:07 -08:00
1364b2af17
Fixed array concat if null ( #4975 )
2024-02-08 18:28:24 +00:00
8beb092a33
DENG-2492 ga_sessions_v2 logic updates ( #4988 )
...
* DENG-2492 initial commit
* DENG-2492 reformat script.sql
* DENG-2492 fix typos
* switch out DL types and fix a typo
* DENG-2492
* DENG-2492 update query logic
* DENG 2492 add meaningful aliases
* DENG-2492 rename rnk to rownum to be more clear
* DENG-2492 add AS for aliasing
* DENG-2492 switch to using
* DENG-2492 move row number out of select and keep only in qualify
* DENG-2492 remove unnecessary a aliases
2024-02-08 12:20:09 -06:00
e056243587
Bump google-cloud-bigquery-storage[fastavro] from 2.23.0 to 2.24.0 ( #4761 )
...
Bumps [google-cloud-bigquery-storage[fastavro]](https://github.com/googleapis/python-bigquery-storage ) from 2.23.0 to 2.24.0.
- [Release notes](https://github.com/googleapis/python-bigquery-storage/releases )
- [Changelog](https://github.com/googleapis/python-bigquery-storage/blob/main/CHANGELOG.md )
- [Commits](https://github.com/googleapis/python-bigquery-storage/compare/v2.23.0...v2.24.0 )
---
updated-dependencies:
- dependency-name: google-cloud-bigquery-storage[fastavro]
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-02-08 09:41:53 -06:00
37af26efdf
Bump gitpython from 3.1.40 to 3.1.41 ( #4802 )
...
Bumps [gitpython](https://github.com/gitpython-developers/GitPython ) from 3.1.40 to 3.1.41.
- [Release notes](https://github.com/gitpython-developers/GitPython/releases )
- [Changelog](https://github.com/gitpython-developers/GitPython/blob/main/CHANGES )
- [Commits](https://github.com/gitpython-developers/GitPython/compare/3.1.40...3.1.41 )
---
updated-dependencies:
- dependency-name: gitpython
dependency-type: direct:production
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Anna Scholtz <anna@scholtzan.net>
2024-02-08 09:41:29 -06:00
dab99e9a4b
Bump symbolic from 12.4.1 to 12.8.0 ( #4820 )
...
Bumps [symbolic]() from 12.4.1 to 12.8.0.
---
updated-dependencies:
- dependency-name: symbolic
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Anna Scholtz <anna@scholtzan.net>
2024-02-08 09:40:43 -06:00
180db53c86
Bump rich-click from 1.7.2 to 1.7.3 ( #4992 )
...
Bumps [rich-click](https://github.com/ewels/rich-click ) from 1.7.2 to 1.7.3.
- [Release notes](https://github.com/ewels/rich-click/releases )
- [Changelog](https://github.com/ewels/rich-click/blob/main/CHANGELOG.md )
- [Commits](https://github.com/ewels/rich-click/compare/v1.7.2...v1.7.3 )
---
updated-dependencies:
- dependency-name: rich-click
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-02-08 09:39:37 -06:00
dd96eea480
Adding daily country and channel counts for fenix meta attribution data ( #4806 )
...
* Adding daily country and channel counts for fenix meta attribution data
Co-authored-by: Anna Scholtz <anna@scholtzan.net>
* Removing extraneous field
---------
Co-authored-by: Rowan Vasquez <rvasquez@XD91WLTP7Q.lan>
Co-authored-by: Frank Bertsch <frank.bertsch@gmail.com>
Co-authored-by: Anna Scholtz <anna@scholtzan.net>
2024-02-07 18:15:28 -05:00
373fa54c61
Deng 2579 gclid conversions v2 creation ( #4972 )
...
* initial commit of code for gclid_conversions_v2
* change owners to Marlene and Katie
* add gclid_conversions_v2 to Access Denied bqetl_project.yaml
2024-02-07 14:03:32 -08:00
378ce60be3
Added user type to view ( #4986 )
2024-02-07 13:36:33 -08:00
4d7c2cf0bf
DS-3102 Add checks to clients_first_seen_v2 ( #4982 )
...
* DS-3102 Add checks to clients_first_seen_v2
* Formatting
* Fully qualify and template source tables
* Update to countif instead of assert not exists
* Add AS table aliases
2024-02-07 15:55:30 -05:00
d980b3b211
DENG-2492 - update logic of new ga_sessions_v2 table ( #4985 )
...
* DENG-2492 - update logic of new ga_sessions_v2 table
* DENG-2492 - update logic of new ga_sessions_v2 table
2024-02-07 14:54:18 -06:00
802da71a2c
Add ETLs and views for Google Search Console data (DENG-1733) ( #4892 )
...
* Add ETLs for historical Google Search Console data synced by Fivetran.
* Fix formatting of `CASE` subclauses like `WHEN` inside Jinja blocks.
* Add ETLs for current Google Search Console data exported directly to BigQuery.
* Add views for Google Search Console data.
2024-02-07 12:53:32 -08:00
0b8e8f14a4
Push tags for commits on generated-sql branch ( #4984 )
...
* Push tags for commits on generated-sql branch
* Update .circleci/config.yml
Co-authored-by: Alexander <anicholson@mozilla.com>
---------
Co-authored-by: Alexander <anicholson@mozilla.com>
2024-02-07 11:07:32 -08:00
e10b93c06c
fix typo in CI ( #4983 )
...
Typo in circleci
2024-02-07 10:08:56 -08:00
df98c38e6c
Replace the `mozilla_vpn_derived.vat_rates_v1` ETL with a CSV file (bug 1878898) ( #4976 )
...
* Replace the `mozilla_vpn_derived.vat_rates_v1` ETL with a CSV file containing the same data.
The source Google Sheet for the `mozilla_vpn_derived.vat_rates_v1` ETL has apparently been deleted (bug 1878898).
* Add missing VAT rates for one VPN wave 3 country.
* Add missing VAT rates for seven VPN wave 6 countries.
2024-02-07 09:48:31 -08:00
07b748849d
Update `country_codes_v1` regions from latest UN Statistics Division data. ( #4978 )
2024-02-07 09:30:22 -08:00
fa1c2492bd
Use caching in generate-sql ( #4953 )
...
* Use caching in generate-sql
* Make change in sql generator
* Remove file
* Add CI comments
2024-02-07 08:57:05 -08:00
e2f33ed29b
Add debug messages for `gcp.json` file. ( #4979 )
2024-02-07 08:12:25 -08:00
ef7bd6f34f
Bump pathos from 0.3.1 to 0.3.2 ( #4980 )
...
Bumps [pathos](https://github.com/uqfoundation/pathos ) from 0.3.1 to 0.3.2.
- [Release notes](https://github.com/uqfoundation/pathos/releases )
- [Commits](https://github.com/uqfoundation/pathos/compare/pathos-0.3.1...0.3.2 )
---
updated-dependencies:
- dependency-name: pathos
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-02-07 10:29:50 +01:00
3bc79e6745
DENG-2492 - update time on site to convert to seconds, add comments ( #4970 )
...
* DENG-2492 - update time on site to convert to seconds, add additional comments to code
* DENG-2492 update formatting of updated SQL file
* DENG-2492 update time_on_site to be int64
* DENG-2492 update query formatting
2024-02-06 15:21:38 -06:00
0a9ee2434a
AD-178 Update event_aggregates tables to pull query_type from the Ads team's table ( #4967 )
...
* Pull query_type from adm.blocks table
* Do the same in event_aggregates_v1
* Update tests
2024-02-06 13:37:45 -05:00
ab68b6fa04
events stream: Convert nested maps in metrics ( #4964 )
2024-02-06 14:21:00 +01:00
d192f9a627
Bump cryptography from 41.0.4 to 42.0.0 ( #4963 )
...
Bumps [cryptography](https://github.com/pyca/cryptography ) from 41.0.4 to 42.0.0.
- [Changelog](https://github.com/pyca/cryptography/blob/main/CHANGELOG.rst )
- [Commits](https://github.com/pyca/cryptography/compare/41.0.4...42.0.0 )
---
updated-dependencies:
- dependency-name: cryptography
dependency-type: indirect
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-02-06 10:47:33 +01:00
5d0c30c84a
Fb deng 2492 ga sessions v2 ( #4962 )
...
* DENG-2492 - update logic for new ga_sessions_v2 load process
* DENG-2492 change update process
2024-02-05 16:23:17 -06:00
f7648d32fe
DENG-2492 - Create new GA4 derived table ga_sessions_v2 ( #4961 )
...
* DENG-2492 - initial commit
* DENG-2492 rename query.sql to script.sql
2024-02-05 15:29:26 -06:00
d43d6ebc23
removed view ( #4932 )
2024-02-05 12:04:08 -08:00
013c63a600
Add moso datasets with DBT workgroup access ( #4960 )
...
Co-authored-by: Wesley Dawson <whd@mozilla.com>
2024-02-05 18:50:17 +00:00
138841d351
Package bqetl and publish to PyPI ( #4917 )
...
* pyproject.toml for bqetl
* Correctly resolve SQL generators from package
* CircleCI config to publish tagged versions to PyPI
* Get version from git tags
2024-02-05 09:04:04 -08:00
a4c7b0ab40
Remove gke_command usages ( #4900 )
2024-02-05 08:40:34 -08:00
95ddc96deb
adding missing field in the final group by ( #4958 )
2024-02-05 17:32:19 +01:00
89aff17297
DS-3104 Create version 2 of Clients Last Seen ( #4236 )
...
* DS-3104. Create quer, metadata and schedule clients_last_seen_v2. Update view clients_last_seen to use this version.
* Update metadata and formatting
* Add to dry-run skip
* Update metadata
---------
Co-authored-by: Alexander Nicholson <anicholson@mozilla.com>
2024-02-05 10:38:49 -05:00
a56df852cc
added submission_date and made it the partition key to address rows belonging to different partitions error ( #4956 )
2024-02-05 14:49:51 +01:00
e38355787b
Fix main_summary_active_addons to handle additional addon fields ( #4955 )
2024-02-04 09:44:56 -08:00
fa4eeaefc4
Update ios feature usage metrics ( #4950 )
...
* update ios feature_usage_metrics
* update ios feature_usage_metrics
* Update query.sql
* Update query.sql
2024-02-02 13:44:22 -08:00
3084f7ed88
DENG-2492 - create new GA4 www_site_page_metrics_v2 ( #4952 )
...
* DENG-2492 - create new GA4 www_site_page_metrics_v2
* DENG-2492 fix query formatting
2024-02-02 12:37:54 -06:00
9788ff7293
feat(DENG-2481): Add fenix install referrer to fenix firefox android clients ( #4940 )
...
* Tweaking firefox_android_clients_v1 to also include play_store attribution fields
* removed additional logic used for testing found within _previous CTE
* removed firefox_android_clients_v1 init.sql in favour of templating via is_init() inside the query
* Made changes as suggested by fbertsch in PR#4940
* Fixing sql tests
2024-02-02 17:42:52 +01:00
3560ac46da
DENG-2492 - add new column engagement time and update hit time column… ( #4947 )
...
* DENG-2492 - add new column engagement time and update hit time column definition
* DENG-2492 - fix formatting for updated SQL
* DENG-2492 update hit_time calculation
2024-02-01 14:57:53 -06:00
acfbcbfbea
Revert "Update clients_last_seen_v1 and clients_first_seen to clients_first_s…" ( #4948 )
...
This reverts commit 39a89a95ea
2024-02-01 15:30:27 -05:00
045f08877b
Add schema.yaml files for event_aggregates and event_aggregates_suggest views ( #4946 )
2024-02-01 13:36:11 -05:00
6971439f23
DENG-2492 - update column desc to accurately reflect new column logic ( #4945 )
2024-02-01 11:46:11 -06:00
f36e75ab2b
Revert "Restrict derived view schema generation to views with upstream schema…" ( #4941 )
...
This reverts commit f5ee129b63
2024-02-01 12:08:51 -05:00
e7c7651935
GA4 bounce logic updates - www_site_hits_v2 and www_site_landing_page_metrics_v2 ( #4944 )
...
* DENG-2492-updating bounce logic in www_site_hits_v2
* Updating bounce logic to match GA4 new definition
2024-02-01 10:41:37 -06:00
f95e1010fd
Add schema files for event_aggregates and event_aggregates_suggest ( #4943 )
2024-02-01 11:14:29 -05:00
rzhao
kik-kik
Sean Rose
Katie Windau
whd
Anna Scholtz
Winnie Chan
dependabot[bot]
Rowan
Marlene Hirose
Alexander
Curtis Morales
Jan-Erik Rediger
Chelsey Beck
Lucia