c281400486
Enforce isort via pytest ( #3384 )
2022-11-30 11:45:05 -08:00
c2e9232162
Bump maven-dependency-plugin from 3.3.0 to 3.4.0 ( #3380 )
2022-11-30 17:41:54 +00:00
b62df40396
feat(DENG-351): fxa_users_services_devices_daily_v1 model added to firefox_accounts_derived dataset ( #3378 )
...
* fxa_users_services_devices_daily_events_v1 model added to firefox_accounts_derived dataset
* added schema.yaml for fxa_users_services_devices_daily_events_v1
* removed invalid comment from metadata.yaml for fxa_users_services_devices_daily_events_v1
* removed invalid comment from metadata.yaml for fxa_users_services_devices_daily_events_v1
* added bigquery table configuration
* added more comments as requested by @gkabbz in PR#3378
* regenerated dags/bqetl_fxa_events.py
* made some tweaks to the query to make it more explicit
* fixed fxa_events DAG
* fixed fxa_users_services_devices_daily_events_v1 schema
* updated table description for fxa_users_services_devices_daily_events_v1
* renamed fxa_users_services_devices_daily_events_v1 to fxa_users_services_devices_daily_v1
2022-11-30 14:16:07 +00:00
6371122594
RS-392 - Added new profiles and activitysegment via unified_metrics ( #3375 )
...
* Added new profiles and activitysegment via unified_metrics
* Updated client aggregations
* submission_date typo
* re-generate unified DAG
* formatting
2022-11-29 10:18:17 -05:00
091f5d8109
letting through some of ios sync events in the fxa_auth_events query ( #3377 )
2022-11-28 10:46:28 +00:00
3f8add3cfe
removed is_deleted as this field is not being generated/used ( #3321 )
2022-11-25 14:18:43 +00:00
b642816724
feat(DENG-537): tweaked fxa queries to support partitioned tables instead of sharded tables - STAGING ( #3371 )
...
* tweaked staging fxa queries to support partitioned tables instead of sharded tables
* updated table reference to the new table
2022-11-24 16:19:17 +00:00
ab852cbd66
[DENG-319 / DSRE-1029] access for views ( #3376 )
...
* bqetl query create
* Update query.sql
* bqetl query validate
* dag
* daily only
* fix: sanitized, not unsanitized
* fix: add owners & workgroup to metadata.yaml
* Update bqetl_search_terms_daily.py
* Update query.sql
* fix: daily to weekly
* Revert "fix: daily to weekly"
This reverts commit 7fb4afab5cb7388d4d42
2022-11-23 18:08:50 +00:00
be655cd7a1
Make ./bqetl query schema deploy work for Python scripts without --force ( #3374 )
...
* Fixes #3368 - Make schema deploy work without --force
* fix format issues
* fix the cli/query.py
* fix black format issues
2022-11-23 10:12:11 -05:00
bdacb952a2
DENG-319 ( #3216 )
...
* bqetl query create
* Update query.sql
* bqetl query validate
* dag
* daily only
* fix: sanitized, not unsanitized
* fix: add owners & workgroup to metadata.yaml
* Update bqetl_search_terms_daily.py
* Update query.sql
* fix: daily to weekly
* Revert "fix: daily to weekly"
This reverts commit 7fb4afab5cb7388d4d42
2022-11-22 13:17:33 -08:00
9e09d4a969
Created dervied table for big query usage and tables reporting ( #3357 )
2022-11-21 10:23:04 -08:00
9dd62cbc16
Add namespace to experimenter_experiments ( #3372 )
...
* feat: add enrollment end date
* fix: bug in status
* feat: add targeted percent
* feat: add namespace
2022-11-21 12:02:53 -06:00
9b49c78953
Filter out 'Focus Android Glean BrowserStack' data in mozdata.unified_metrics. ( #3360 )
...
Co-authored-by: Lucia Vargas <lvargas@mozilla.com>
2022-11-21 16:44:54 +01:00
50a69987c5
Add targeted percent to experimenter_experiments ( #3369 )
...
* feat: add enrollment end date
* fix: bug in status
* feat: add targeted percent
* Update sql/moz-fx-data-experiments/monitoring/experimenter_experiments_v1/query.py
Co-authored-by: Anna Scholtz <anna@scholtzan.net>
Co-authored-by: Anna Scholtz <anna@scholtzan.net>
2022-11-21 09:18:49 -06:00
52e4def227
Add enrolllment end date to experimenter_experiments ( #3366 )
...
* feat: add enrollment end date
* fix: bug in status
2022-11-17 13:29:21 -06:00
dfc2b516c8
Update sql/moz-fx-data-experiments/monitoring/experimenter_experiments_v1/query.py
...
Co-authored-by: Anna Scholtz <anna@scholtzan.net>
2022-11-17 08:58:08 -08:00
0d339778ff
Update sql/moz-fx-data-experiments/monitoring/experimenter_experiments_v1/query.py
...
Co-authored-by: Anna Scholtz <anna@scholtzan.net>
2022-11-17 08:58:08 -08:00
1484397c93
feat: add targeting
2022-11-17 08:58:08 -08:00
a05e0dd80c
Update event_aggregates_v1 tests
2022-11-17 08:34:36 -08:00
bfa2b26fb1
feat: add filter to 2 more queries
2022-11-17 08:34:36 -08:00
64f808652c
Add check to contextual_services_derived.event_aggregates_v1 to prevent backfills on deleted partitions
2022-11-17 08:34:36 -08:00
27fbf59ba0
Ensure target path exists when generating country_code_lookup ( #3362 )
2022-11-17 10:59:41 +01:00
f1afd93726
removed device_id duplication in fxa_log_auth_events query ( #3361 )
2022-11-16 16:55:34 +01:00
e7cf0dc09e
making sure device_id is hacked in fxa_log_auth_events and removed from fxa_log_content_events as the source table does not have this field ( #3358 )
2022-11-16 16:07:05 +01:00
35897a6b1e
Updates to tiles clients daily ( #3355 )
...
* Fix nulls
* Sample last repeated row from Android + iOS data
* Update sponsored_tiles_clients_daily
Client IDs difficult to match between the pings, especially for mobile. This solution overcounts clients in exchange for more accurate event counts
* Update to pull basic info from unified_metrics
* Remove Android metrics ping data
tiles enabled at startup metric
* Update DAGs
* Format SQL
* Update schema file
Co-authored-by: Curtis Morales <cmorales@mozilla.com>
2022-11-15 19:58:41 -05:00
52eeb88d60
Change the ownership of remotesettings_suggestions ( #3353 )
...
I'm no longer working on this and it might be wiser for folks more deep-in into the topic to own it!
I believe @chelseatroy and @quiiver are probably the right folks (or know who the right folks should be!)
Co-authored-by: whd <whd@users.noreply.github.com>
2022-11-15 13:43:56 -08:00
1a5bf3c9e7
Feat/adding device id to fxa data ( #3314 )
...
* added fxa_users_services_devices_first_seen_v1 model and adjusted models to propagate device_id
* reverted sql/moz-fx-data-shared-prod/firefox_accounts_derived/funnel_events_source_v1/query.sql to main branch version
* implemented changes as suggested by @relud in PR#3314
* fixed missing )
* added device_id to fxa_auth_bounce_events table
* applied sql formatting
2022-11-15 21:03:42 +01:00
a4ef437440
Revert "Created dervied table for big query usage and tables reporting ( #3350 )" ( #3356 )
...
This reverts commit 3730698355
2022-11-15 09:22:08 -08:00
3730698355
Created dervied table for big query usage and tables reporting ( #3350 )
...
* Added bq usage dataset
* Added bq tables inventory dataset
* Added bq table storage dataset
* Added bigquery table storage timeline daily dataset
* Updated bqetl monitoring dag
2022-11-15 09:02:47 -08:00
8327a67e81
[ADPRODUCTS-367] replace NULL with FALSE ( #3342 )
...
* fix: fill NULL with FALSE
* fix: remove profile age
* fix: add country
* Update schema file
Co-authored-by: Curtis Morales <cmorales@mozilla.com>
2022-11-14 13:46:09 -05:00
1897ec9c05
Add M1 Mac setup guide ( #3302 )
2022-11-14 11:03:32 -05:00
1ab5158959
Add product name to VPN aggregate tables (DS-2185) ( #3348 )
2022-11-09 12:59:20 -08:00
8a3f146e9e
DENG-434 Add focus android data from Glean to unified metrics ( #3316 )
...
* DENG-434. Update cohorts queries to use the unified_metrics view instead of the table.
* DENG-434. Make columns (is_default_browser, uri_count) available after updating the sql generators in https://github.com/mozilla/bigquery-etl/pull/3315 .
* DENG-434. Document the change to use app name convention with suffix ' Glean'
* Update sql/moz-fx-data-shared-prod/telemetry/unified_metrics/metadata.yaml
* Update sql/moz-fx-data-shared-prod/telemetry/unified_metrics/metadata.yaml
Co-authored-by: Lucia Vargas <lvargas@mozilla.com>
Co-authored-by: Alekhya <88394696+alekhyamoz@users.noreply.github.com>
2022-11-09 12:37:44 -05:00
323fb96ecb
Tweaking Apple Silicon comment to red to make it more prominent ( #3346 )
2022-11-09 16:45:18 +01:00
789c13d386
Include VPN+Relay bundle subscriptions in VPN ETL (DS-2185) ( #3345 )
2022-11-08 15:41:24 -08:00
d93ba70c94
Added pocket_available_on_newtab and mozilla_vpn_available to countri… ( #3332 )
...
* Added pocket_available_on_newtab and mozilla_vpn_available to countries data
2022-11-08 16:36:21 -05:00
38578a5f08
Handle Stripe subscription plan changes (DS-2160) ( #3289 )
2022-11-08 10:45:35 -08:00
7df38ebda4
DENG-453 add isp to unified metrics ( #3333 )
...
* DENG-453 Make ISP available in unified_metrics
* Update schema.yaml
Add column `isp` to table `unified_metrics`
* fix tests
Co-authored-by: Alekhya Kommasani <akommasani@mozilla.com>
Co-authored-by: Alekhya <88394696+alekhyamoz@users.noreply.github.com>
2022-11-08 13:19:41 -05:00
cd224645f2
updated badge status url ( #3344 )
2022-11-08 18:07:04 +01:00
77a1d3c7e2
[ADPRODUCTS-367] fill NAs with False ( #3340 )
...
* fix: fill NULL with FALSE
2022-11-07 16:54:15 -05:00
1c415bec38
Accept --dry-run as a flag in `bqetl query` commands ( #3339 )
2022-11-07 16:35:10 -05:00
803cd21bce
[ADPRODUCTS-367] Fix source table & advertisers ( #3341 )
...
* fix: correct clicks table
* fix: standardize advertiser
* fix: missing commas
* Update sql/moz-fx-data-shared-prod/contextual_services_derived/event_aggregates_spons_tiles_v1/query.sql
Co-authored-by: Curtis Morales <cmorales@mozilla.com>
* Update sql/moz-fx-data-shared-prod/contextual_services_derived/event_aggregates_spons_tiles_v1/query.sql
Co-authored-by: Curtis Morales <cmorales@mozilla.com>
Co-authored-by: Curtis Morales <cmorales@mozilla.com>
2022-11-07 15:48:33 -05:00
b980d2b17d
fix tests ( #3336 )
2022-11-07 09:45:00 -05:00
4cedc2b127
Fix #3303 : Add support to backfill monthly partition tables ( #3337 )
2022-11-04 23:05:16 -04:00
a365140fef
[ADPRODUCTS-367] Updates to tiles clients daily ( #3338 )
...
* Fix nulls
* Sample last repeated row from Android + iOS data
2022-11-04 14:56:38 -07:00
6bdc814d38
separate the test windows for mean and range data validation checks ( #3331 )
...
Co-authored-by: whd <whd@users.noreply.github.com>
2022-11-04 10:34:18 -07:00
e98684cb0e
Update metadata on every schema deploy ( #3334 )
2022-11-04 12:32:03 -04:00
9d650106d6
DENG-452. Update the query template for mobile to set the correct que… ( #3328 )
...
* DENG-452. Update the query template for mobile to set the correct query for uri_count and is_default_browser.
* Update metrics_templating.yaml
Add space.
* Removing placeholder comment in template.
Co-authored-by: Lucia Vargas <lvargas@mozilla.com>
2022-11-03 10:38:49 +01:00
7739b1413f
DSRE 1095 ( #3323 )
...
* Revert "Remove dataset workgroup_access for contextual services (#2361 )"
This reverts commit 8b79acb356
2022-11-02 21:56:07 +00:00
b271543332
Remove workaround for probeinfo always returning gzip ( #3330 )
2022-11-02 13:50:51 -07:00
Daniel Thorn
dependabot[bot]
kik-kik
Alexander
Rebecca BurWei
Alekhya
Winnie Chan
Lucia
Rebecca BurWei
Anna Scholtz
skahmann3
Alessio Placitelli
Curtis Morales
Sean Rose
Eduardo Filho
Chelsea Troy
whd