66729aa702
FXA-6721 Setup import of accounts table from FxA production CloudSQL ( #4423 )
2023-10-25 09:50:25 +02:00
ae7d0f4766
Change pocket_available for new Pocket markets ( #4472 )
2023-10-24 12:50:45 -07:00
55c5d412c1
Allow running multiple checks ( #4471 )
...
* Allow running multiple checks
* Don't yield anything on no matches
2023-10-24 14:39:01 -04:00
2b2697e8f3
[RS-826] New job to calculate newtab visits -> activity stream sessions ( #4387 )
...
* New job to calculate newtab visits -> activity stream sessions
* Removing newline chars at end of file
* Removing newline chars at end of file
* Removing newline chars at end of file
* Addressing comment suggestions
* Format
* Add bqetl_ads DAG
* Add ACL to nt_visits_to_sessions_conversion_factors_daily_v1
* Add metadata files
* Add view to dry_run skip list
* Oops, fix the view
---------
Co-authored-by: Curtis Morales <cmorales@mozilla.com>
2023-10-24 12:51:14 -04:00
a4d2a0a2d1
Bump mkdocs from 1.5.2 to 1.5.3 ( #4321 )
...
Bumps [mkdocs](https://github.com/mkdocs/mkdocs ) from 1.5.2 to 1.5.3.
- [Release notes](https://github.com/mkdocs/mkdocs/releases )
- [Commits](https://github.com/mkdocs/mkdocs/compare/1.5.2...1.5.3 )
---
updated-dependencies:
- dependency-name: mkdocs
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-10-24 11:49:36 -04:00
c3e98c341b
Bump urllib3 from 1.26.14 to 1.26.18 ( #4450 )
...
Bumps [urllib3](https://github.com/urllib3/urllib3 ) from 1.26.14 to 1.26.18.
- [Release notes](https://github.com/urllib3/urllib3/releases )
- [Changelog](https://github.com/urllib3/urllib3/blob/main/CHANGES.rst )
- [Commits](https://github.com/urllib3/urllib3/compare/1.26.14...1.26.18 )
---
updated-dependencies:
- dependency-name: urllib3
dependency-type: indirect
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-10-24 10:19:49 -04:00
fb2574cc33
Bump gitpython from 3.1.36 to 3.1.40 ( #4458 )
...
Bumps [gitpython](https://github.com/gitpython-developers/GitPython ) from 3.1.36 to 3.1.40.
- [Release notes](https://github.com/gitpython-developers/GitPython/releases )
- [Changelog](https://github.com/gitpython-developers/GitPython/blob/main/CHANGES )
- [Commits](https://github.com/gitpython-developers/GitPython/compare/3.1.36...3.1.40 )
---
updated-dependencies:
- dependency-name: gitpython
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-10-24 09:01:16 -05:00
5e42214cc0
Bump gcsfs from 2023.6.0 to 2023.10.0 ( #4468 )
...
Bumps [gcsfs](https://github.com/fsspec/gcsfs ) from 2023.6.0 to 2023.10.0.
- [Commits](https://github.com/fsspec/gcsfs/compare/2023.6.0...2023.10.0 )
---
updated-dependencies:
- dependency-name: gcsfs
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-10-24 06:44:38 -07:00
d529f2583d
RS-829 Add Focus to the list of app_names for mobile search revenue levers ( #4394 )
...
* Add Focus to the list of app_names for mobile search revenue levers
* Add country for Bing and DDG
2023-10-23 16:17:44 -04:00
e1b5a2019a
Update clients_first_seen_28_days_later metadata to write to correct partition ( #4466 )
2023-10-23 15:03:48 -04:00
8c499c5772
DENG-476 - Remove telemetry.main et al view overrides so that they will point at v5 ( #4456 )
2023-10-23 10:05:26 -07:00
f99de5d9e5
FXA-6721 Add user-facing view for stage FxA accounts table ( #4413 )
2023-10-23 16:16:14 +02:00
a685e17279
CRemove the filter on messaging_system_event ( #4465 )
2023-10-23 09:53:10 -04:00
a72ffc4dab
Remove ETL checks ( #4460 )
...
Co-authored-by: akkomar <akkomar@users.noreply.github.com>
2023-10-23 08:59:18 -04:00
c5dd137f26
Fix error_aggregates sample data references ( #4462 )
2023-10-19 15:43:47 -04:00
2eabb130ca
DENG-1714 - Create clients_first_seen_28_days_later ( #4395 )
...
* DENG-1714 - Create clients_first_seen_28_days_later
* Regenerate DAG
2023-10-19 14:14:45 -04:00
ac0af012c2
Add opt-in to running checks for backfill ( #4455 )
2023-10-18 17:34:58 -04:00
6f3d34ba67
DS3244 - Add derived datasets for review checker data ( #4447 )
...
* Add review checker derived datasets
* Add bqetl_review_checker dag
Fix
* Fix CI validate dag step
* Incorporate feedback from Alex
* Fix CI
* change client last seen to clients first seen
change client last seen to clients first seen
* fix dag
2023-10-18 16:57:15 -04:00
cd0f042de5
feat(DENG-1777): adding retention tables for fenix ( #4425 )
...
* Adding retention tables for firefox android / fenix
* regenerated bqetl_analytics_tables DAG
* fixing schema for fenix.funnel_retention_clients_week_2_v1
* fixed incorrect wrong table source for retention week 4 and format SQL applied
2023-10-18 11:56:42 -04:00
7a36416554
Set project in init jobs ( #4453 )
...
This fixes https://github.com/mozilla/bigquery-etl/pull/4452
2023-10-18 16:58:18 +02:00
0171f93596
Set project in init jobs ( #4452 )
2023-10-18 16:04:09 +02:00
c3c5ecffd4
Don't set destination table for init jobs ( #4451 )
...
This reverts https://github.com/mozilla/bigquery-etl/pull/4193/files
By convention all but two init.sql jobs use `CREATE TABLE` statement. Setting destination table on a job that runs these queries causes an `BadRequest: 400 Cannot set destination table in jobs with DDL statements` error as observed in [1].
Apart from removing setting of destination_table this fixes two init queries.
[1] https://workflow.telemetry.mozilla.org/dags/copy_deduplicate/grid?dag_run_id=scheduled__2023-10-17T01%3A00%3A00%2B00%3A00&task_id=baseline_clients_first_seen&tab=logs
2023-10-18 14:45:22 +02:00
3792797d1d
DENG-476 - Update clients daily and clients first seen to reference main_v5 ( #4438 )
2023-10-17 16:54:42 -07:00
164ba19abf
Glean usage checks ( #4445 )
...
* WIP: Add checks for glean_usage
* Ignore pycache in autogenerated click cmds
* Move check to backfill command
* Remove view checks
2023-10-17 17:03:41 -04:00
34c8cf35e7
Fix error_aggregates tests to reference main_v5 ( #4448 )
2023-10-17 13:16:40 -07:00
7f0b7d522a
DENG1546 - Fix bqetl_serp DAG failure ( #4429 )
2023-10-17 15:51:48 -04:00
36cd455d2a
Add `json.from_map` UDF ( #4414 )
2023-10-17 14:00:25 -05:00
358ea9d574
Fix Glean usage generation; don't skip query generation when tables don't exist ( #4446 )
2023-10-17 10:54:57 -07:00
b692735c80
DENG-476 - Update FoG decision ETL to reference main_v5 ( #4436 )
2023-10-17 07:14:10 -07:00
40ffe7f0b6
DENG-476 - Update UDF to reference main pings instead of main_v4 ( #4440 )
2023-10-16 15:49:58 -07:00
3b14b881f8
DENG-476 - Remove unnecessary references to main_v4 in addons ETL ( #4434 )
2023-10-16 15:13:37 -07:00
065f4c78f9
DENG-476 - Update latest versions to reference main_v5 ( #4441 )
2023-10-16 14:56:43 -07:00
b63779faf1
DENG-476 - Update error aggregates to reference main_v5 ( #4439 )
2023-10-16 14:40:12 -07:00
167d3a7a31
DENG-476 - Remove unnecessary reference to main_v4 from ssl ratios ( #4437 )
2023-10-16 14:17:25 -07:00
f0c53d7afe
DENG1546 - Fix bqetl serp DAG failure due to missing parameter ( #4426 )
...
* DENG1546 - Fix DAG failure due to missing parameter
* Fix CI issue
* Fix logic in parameter
2023-10-16 13:23:57 -04:00
f0af9885c4
DENG1546 - Fix BQ partition issue ( #4419 )
...
* DENG1546 - Fix BQ partition issue
FIX CI issue
fix CI issue
* Update sql_generators/serp_events/templates/metadata.yaml
Co-authored-by: Anna Scholtz <anna@scholtzan.net>
* Update sql_generators/serp_events/templates/metadata.yaml
Co-authored-by: Anna Scholtz <anna@scholtzan.net>
* Update sql_generators/serp_events/templates/metadata.yaml
Co-authored-by: Anna Scholtz <anna@scholtzan.net>
* fix dag generation
---------
Co-authored-by: Anna Scholtz <anna@scholtzan.net>
2023-10-14 13:12:44 -04:00
4728884e3c
DS-2642 - Update tax address source in stripe payout report ( #4422 )
2023-10-13 13:16:12 -07:00
f41c85a37d
DS-2642 - Add tax amount to stripe payout report ( #4421 )
2023-10-13 11:56:00 -07:00
4bbbc32a5b
Put assert UDFs in `mozfun` project ( #4367 )
...
* Put assert UDFs in `mozfun` project.
* Tweak syntax in `assert.array_equals()` to avoid SQLGlot parsing error.
https://github.com/tobymao/sqlglot/issues/2348
* Fix SQL syntax error in `assert.struct_equals()` tests.
* Fix UDF dependency file path logic when deploying to stage.
* Change regular expressions in `parse_routine` module to allow quotes around routines' dataset and name.
2023-10-13 10:58:42 -07:00
95e90efb4d
fix(ci): Manually set SSH configuration for multiple SSH keys. ( #4420 )
2023-10-13 11:46:23 -04:00
109b9f0835
feat(DENG-1599): added ETL checks related to firefox_ios_derived.new_profile_activation_v2 ( #4411 )
...
* added ETL checks related to firefox_ios_derived.new_profile_activation_v2
* regenerated bqetl_firefox_ios DAG
2023-10-13 16:42:02 +02:00
83569d8211
Add sampling to glam-fog ( #4409 )
...
* Add sampling to glam-fog
* Simplify count logic
* Update bigquery_etl/glam/templates/clients_daily_histogram_aggregates_v1.sql
Co-authored-by: Anna Scholtz <anna@scholtzan.net>
* Update bigquery_etl/glam/templates/clients_daily_scalar_aggregates_v1.sql
Co-authored-by: Anna Scholtz <anna@scholtzan.net>
---------
Co-authored-by: Anna Scholtz <anna@scholtzan.net>
2023-10-13 09:24:04 -04:00
22c44fc62a
DENG1546 -Add a derived dataset for serp events ( #4407 )
...
* Add a derived dataset for serp events
* Fix CI issues
Add bqetl_dag
Fix CI issues
* reapply missing updates
* Fix the query logic
Fix the query logic
* Fix sql query
* fix partition field
2023-10-12 19:38:29 -04:00
148d4eb539
feat: Cirrus monitor query ( #4393 )
...
* feat: Cirrus monitor query
* Update sql_generators/experiment_monitoring/templates/experiments_daily_active_clients_v1/query.sql
Co-authored-by: Mike Williams <102263964+mikewilli@users.noreply.github.com>
* feat: Monitor cirrus events
* fix: Use events
* Update sql/moz-fx-data-shared-prod/telemetry_derived/experiments_daily_active_clients_v1/query.sql
Co-authored-by: Anna Scholtz <anna@scholtzan.net>
* Update sql_generators/experiment_monitoring/templates/experiment_enrollment_aggregates_v1/query.sql
Co-authored-by: Anna Scholtz <anna@scholtzan.net>
* Update sql_generators/experiment_monitoring/templates/experiment_events_live_v1/init.sql
Co-authored-by: Anna Scholtz <anna@scholtzan.net>
* Update sql_generators/experiment_monitoring/templates/experiment_search_events_live_v1/init.sql
Co-authored-by: Anna Scholtz <anna@scholtzan.net>
* Update sql_generators/experiment_monitoring/templates/experiments_daily_active_clients_v1/query.sql
Co-authored-by: Anna Scholtz <anna@scholtzan.net>
* Update sql_generators/experiment_monitoring/templates/experiments_daily_active_clients_v1/query.sql
Co-authored-by: Anna Scholtz <anna@scholtzan.net>
* Update sql_generators/experiment_monitoring/templates/templating.yaml
Co-authored-by: Anna Scholtz <anna@scholtzan.net>
* Update sql_generators/experiment_monitoring/templates/templating.yaml
Co-authored-by: Anna Scholtz <anna@scholtzan.net>
* Update sql/moz-fx-data-shared-prod/monitor_cirrus_derived/experiment_search_events_live_v1/init.sql
Co-authored-by: Anna Scholtz <anna@scholtzan.net>
* Update sql/moz-fx-data-shared-prod/monitor_cirrus_derived/experiment_events_live_v1/init.sql
Co-authored-by: Anna Scholtz <anna@scholtzan.net>
* feat: Update view to table
* feat: Update view to table
* use live tables instead of stable for live events
---------
Co-authored-by: Mike Williams <102263964+mikewilli@users.noreply.github.com>
Co-authored-by: Anna Scholtz <anna@scholtzan.net>
Co-authored-by: Mike Williams <mwilliams@mozilla.com>
2023-10-12 14:46:26 -07:00
9a6d9d15f4
fix(ci): sync-dags-repo uses wrong branch for private-bigquery-etl ( #4417 )
2023-10-12 17:21:42 -04:00
35ae323487
Funnel generators POC ( #4390 )
...
* Add funnel generation logic
* Example funnel config
* Fix funnel columns
* funnel generation dimensions
* Optimize segmenting generated funnels
* Add funnel generation docs
* Schedule generated funnels
* Skip DAGs with no tasks
* Add background info funnel generator
* Add funnel generation tests
* Fix join_previous_step_on
* Add funnel example config
2023-10-12 14:05:08 -07:00
27a99ca947
fix(ci): Add missing SSH key in sync-dags-repo step ( #4416 )
2023-10-12 16:46:21 -04:00
61da5cca03
Respect sql_dir in dryrun skip ( #4334 )
...
* Respect sql_dir in dryrun skip
* Update bigquery_etl/dryrun.py
Co-authored-by: Sean Rose <1994030+sean-rose@users.noreply.github.com>
* Update bigquery_etl/dryrun.py
Co-authored-by: Sean Rose <1994030+sean-rose@users.noreply.github.com>
* Set sql_dir when using Schema.from_query_file()
---------
Co-authored-by: Sean Rose <1994030+sean-rose@users.noreply.github.com>
2023-10-12 13:27:54 -07:00
c187aeba06
chore(ci): update private-bigquery-etl git submodule in telemetry-airflow-dags repo ( #4415 )
2023-10-12 15:45:54 -04:00
a5bcebc308
Add line numbers for clients_daily aggs ( #4412 )
2023-10-12 12:36:50 -04:00
akkomar
m-d-bowerman
Frank Bertsch
Sergio E. Betancourt
dependabot[bot]
Curtis Morales
Alexander
Daniel Thorn
Alekhya
kik-kik
Anna Scholtz
wil stuckey
Sean Rose
Mikaël Ducharme
Eduardo Filho
Yashika Khurana