1ffbd574ea
CLI backfill command
2020-08-27 14:48:32 -07:00
256b3cb648
CLI query info
2020-08-27 14:48:32 -07:00
5f9f4e9027
CLI dryrun command
2020-08-27 14:48:32 -07:00
d8f782dc62
Add scripts for backfilling and exporting all fenix aggregates ( #1255 )
...
* Add scripts for backfilling and exporting all fenix aggregates
* Update script/glam/export_glean_all_fenix
Co-authored-by: Ben Wu <benjaminwu124@gmail.com>
Co-authored-by: Ben Wu <benjaminwu124@gmail.com>
2020-08-26 11:08:32 -07:00
d518b35575
Fix table name extraction and set write disposition for bigquery_etl_scheduled_queries_cost_v1
2020-08-26 10:20:07 -07:00
bd73aebb6f
Add submission_date to bigquery_etl_scheduled_queries_cost_v1 table
2020-08-26 10:20:07 -07:00
f3b0ab9350
Add dataset for monitoring bigquery-etl query cost
2020-08-26 10:20:07 -07:00
6f30c7feef
Add sample_id to desktop retention view ( #1252 )
...
Co-authored-by: Jeff Klukas <jklukas@mozilla.com>
2020-08-26 09:53:25 -04:00
9eeca4ad5c
Add Fenix, Fx Lite to Day 2-7 Activation ( #1254 )
...
* Add Fenix, Fx Lite to Day 2-7 Activation
Fenix and Firefox Lite were previously filtered from the Day 2-7
Activation metric pending validation in https://jira.mozilla.com/browse/DS-696
* Remove excessive comments
2020-08-25 17:03:49 -04:00
7a069cf408
Allow baseline pings with 'foreground' reason in KPIs ( #1253 )
...
After merging, I will need to backfill these tables back to early April.
DS asked for this change in https://jira.mozilla.com/browse/DS-1018
2020-08-25 10:01:31 -04:00
a5c5cb84ae
CLI secho fix
...
Co-authored-by: Anthony Miyaguchi <amiyaguchi@mozilla.com>
2020-08-24 12:19:18 -07:00
48164652af
CLI allow info for specific DAG
2020-08-24 12:19:18 -07:00
8998d88d50
Add tests for CLI utils
2020-08-24 12:19:18 -07:00
c0fc5b8db2
Add tests for CLI dag command
2020-08-24 12:19:18 -07:00
a81c4b06a3
Refactor CLI utility functions
2020-08-24 12:19:18 -07:00
01585616fa
CLI command for removing DAGs
2020-08-24 12:19:18 -07:00
0289e120e1
CLI command for generating DAGs
2020-08-24 12:19:18 -07:00
6aa1f7aa8d
Add CLI command for creating new DAGs
2020-08-24 12:19:18 -07:00
00d31b73a7
List DAGs
2020-08-24 12:19:18 -07:00
8c1c699771
Fix CLI tests
2020-08-21 11:10:58 -07:00
1d5483afb5
Update DAG generation tests
2020-08-21 11:10:58 -07:00
e1dab7de0f
Add tests for CLI query schedule
2020-08-21 11:10:58 -07:00
cbf560a1fa
Add CLI tests for creating queries
2020-08-21 11:10:58 -07:00
e9d1f16e2c
CLI query schedule allow re-generation
2020-08-21 11:10:58 -07:00
78f3e8f012
CLI query schedule re-generate DAG
2020-08-21 11:10:58 -07:00
e49618fdf5
Add query schedule command to CLI
2020-08-21 11:10:58 -07:00
146a5aebe3
Bug 1660066 - Create view for onboarding user retention analysis in Amplitude ( #1241 )
...
Co-authored-by: Frank Bertsch <fbertsch@mozilla.com>
2020-08-21 13:02:38 -04:00
3715ba87cd
Revert "Use mozfun.norm.fenix_app_info for nondesktop_clients_last_seen ( #1245 )" ( #1249 )
...
This reverts commit 4fe432337e
2020-08-21 10:56:50 -04:00
4fe432337e
Use mozfun.norm.fenix_app_info for nondesktop_clients_last_seen ( #1245 )
...
* Use mozfun.norm.fenix_app_info for nondesktop_clients_last_seen
After this is merged, we can backfill of smoot tables to have GUD
reflect the current and historical app naming schemes.
2020-08-21 09:29:03 -04:00
aa3f827d6f
More complete docs for fenix_app_info ( #1244 )
...
Fixes #1243
2020-08-20 15:42:47 -04:00
6a2d48f037
fenix_app_info documentation fixups ( #1237 )
2020-08-19 08:55:37 -04:00
96b85854d2
Update Glam ETL for Fenix ( #1240 )
...
* Use UNION ALL instead of UNION
* Move tests into separate directory and add test for all fenix products
* Replace channel with *
2020-08-18 16:15:56 -07:00
e72b2777d2
Move experiments_search_aggregates logic to a view ( #1239 )
2020-08-18 15:17:22 -07:00
654401a36d
Fix intermittent verify-dags-up-to-date failures
2020-08-18 15:07:35 -07:00
a92a16c92b
Add mozfun.norm.fenix_app_info UDF ( #1226 )
2020-08-18 16:21:42 -04:00
ceb223e1d8
Use git diff for verifying DAGs in CI
2020-08-18 12:31:38 -07:00
9b8172eb1e
Update integration tests for query scheduling
2020-08-18 12:31:38 -07:00
155e41c734
Sort generated Airflow tasks and dependencies by name
2020-08-18 12:31:38 -07:00
f5c9c4c612
Remove project ID option from generate_airflow_dags script
2020-08-18 12:31:38 -07:00
6d1e00aa42
Verify if generated DAGs are up-to-date
2020-08-18 12:31:38 -07:00
486224ecda
Update experiment_search_aggregates_v1 init destination table
...
Co-authored-by: Daniel Thorn <dthorn@mozilla.com>
2020-08-18 12:23:03 -07:00
75caa0ed36
Add cumulative search counts for experiment search aggregates
2020-08-18 12:23:03 -07:00
f1579104af
Experiments search aggregates live view
2020-08-18 12:23:03 -07:00
4d5b5652e7
experiment_search_aggregates dataset
2020-08-18 12:23:03 -07:00
222e04b081
Fix #1232 - Ignore glam_etl directory when publishing views ( #1234 )
2020-08-18 11:33:50 -07:00
ca2204625d
Add views for logical Fenix app ids in GLAM ETL ( #1221 )
...
* Add views for logical app ids
* Add new generated sql
* Update generate_glean_sql script to handle logical apps
* Update logical app view for partitiontime
* Make sure to generate view for all of the app ids
* Update last versions to be logical app id agnostic
* Add formatting for black
* Fix linting error
* Update bigquery_etl/glam/generate.py
Co-authored-by: Ben Wu <benjaminwu124@gmail.com>
* Add "all" option to STAGE
* Add new metrics added since last PR
Co-authored-by: Ben Wu <benjaminwu124@gmail.com>
2020-08-17 15:05:15 -07:00
c63bd7e3cd
Add decoded stub installer to structured docid count ( #1231 )
2020-08-17 16:57:02 -04:00
a6ef677b7f
Use cached_property for DryRun
2020-08-14 11:08:15 -07:00
47ff0d88f3
Refactor dry run and update integration tests
2020-08-14 11:08:15 -07:00
58b3cf31b7
Add payload_bytes_* tables as referenced_tables for monitoring datasets
2020-08-14 11:08:15 -07:00
Anna Scholtz
Anthony Miyaguchi
Frank Bertsch
Ryan Harter
Jeff Klukas
Nan Jiang
Daniel Thorn
Ben Wu