654401a36d
Fix intermittent verify-dags-up-to-date failures
2020-08-18 15:07:35 -07:00
a92a16c92b
Add mozfun.norm.fenix_app_info UDF ( #1226 )
2020-08-18 16:21:42 -04:00
ceb223e1d8
Use git diff for verifying DAGs in CI
2020-08-18 12:31:38 -07:00
9b8172eb1e
Update integration tests for query scheduling
2020-08-18 12:31:38 -07:00
155e41c734
Sort generated Airflow tasks and dependencies by name
2020-08-18 12:31:38 -07:00
f5c9c4c612
Remove project ID option from generate_airflow_dags script
2020-08-18 12:31:38 -07:00
6d1e00aa42
Verify if generated DAGs are up-to-date
2020-08-18 12:31:38 -07:00
486224ecda
Update experiment_search_aggregates_v1 init destination table
...
Co-authored-by: Daniel Thorn <dthorn@mozilla.com>
2020-08-18 12:23:03 -07:00
75caa0ed36
Add cumulative search counts for experiment search aggregates
2020-08-18 12:23:03 -07:00
f1579104af
Experiments search aggregates live view
2020-08-18 12:23:03 -07:00
4d5b5652e7
experiment_search_aggregates dataset
2020-08-18 12:23:03 -07:00
222e04b081
Fix #1232 - Ignore glam_etl directory when publishing views ( #1234 )
2020-08-18 11:33:50 -07:00
ca2204625d
Add views for logical Fenix app ids in GLAM ETL ( #1221 )
...
* Add views for logical app ids
* Add new generated sql
* Update generate_glean_sql script to handle logical apps
* Update logical app view for partitiontime
* Make sure to generate view for all of the app ids
* Update last versions to be logical app id agnostic
* Add formatting for black
* Fix linting error
* Update bigquery_etl/glam/generate.py
Co-authored-by: Ben Wu <benjaminwu124@gmail.com>
* Add "all" option to STAGE
* Add new metrics added since last PR
Co-authored-by: Ben Wu <benjaminwu124@gmail.com>
2020-08-17 15:05:15 -07:00
c63bd7e3cd
Add decoded stub installer to structured docid count ( #1231 )
2020-08-17 16:57:02 -04:00
a6ef677b7f
Use cached_property for DryRun
2020-08-14 11:08:15 -07:00
47ff0d88f3
Refactor dry run and update integration tests
2020-08-14 11:08:15 -07:00
58b3cf31b7
Add payload_bytes_* tables as referenced_tables for monitoring datasets
2020-08-14 11:08:15 -07:00
d73b21ff6e
Update DAGs
2020-08-14 11:08:15 -07:00
b1a9b4a781
Add dry run tests
2020-08-14 11:08:15 -07:00
0a5adf91ed
Refactor Airflow DAG generation to use dry run
2020-08-14 11:08:15 -07:00
e5c8946171
Dry run get referenced tables
2020-08-14 11:08:15 -07:00
53c856b781
Fix GCS integration test issues
2020-08-13 12:17:23 -07:00
1af07bdb1e
Use qualified table names and udf functions ( #1227 )
...
This makes backfilling inside a sandbox project possible
2020-08-12 16:13:22 -04:00
4bb8eff997
Add distinct docid count queries ( #1225 )
2020-08-11 18:38:42 -04:00
1c71e70838
Bug 1654330 AMO download stats by UTM params ( #1224 )
2020-08-11 13:32:10 -04:00
285efeb991
Add burnham test report table ( #1220 )
...
* Add burnham test report table
For https://github.com/mozilla/burnham/issues/39
* Add test_log_url and test_duration_millis
* Apply suggestions from code review
Co-authored-by: Raphael Pierzina <raphael@hackebrot.de>
Co-authored-by: Raphael Pierzina <raphael@hackebrot.de>
2020-08-10 09:50:27 -04:00
3c6d63739c
Fix formatting.
2020-08-10 09:39:50 -04:00
7298d43c32
Add missing array length call.
2020-08-10 09:39:50 -04:00
b57f24ff1b
Add addon and cert error features
2020-08-10 09:39:50 -04:00
6a83b6223e
Fix mozfun format in generated docs ( #1212 )
2020-08-07 16:35:44 -04:00
5702d96dae
Bug 1646362 - Accessibility inspector derived dataset ( #1171 )
...
Co-authored-by: Jeff Klukas <jklukas@mozilla.com>
2020-08-07 16:27:06 -04:00
4633b5fda4
Provide extra context when dag generation fails due to a malformed query ( #1222 )
2020-08-07 14:40:14 -04:00
5bab4e9d26
Bug 1655146 Add impression_stats_by_experiment
...
This table will be a bit larger than impression_stats_flat, about 1 TB/day
uncompressed. But it will be much more efficient for the class of queries
that are concerned with evaluating performance of a single experiment.
2020-08-07 12:54:10 -04:00
36b7c184e6
Add script to backfill glam tables for a glean product ( #1108 )
...
* Add backfill script for glean products
* Specify product correctly and add target dataset
* Add product to example
* Use datetime.fromisoformat
2020-08-06 15:48:40 -07:00
a58821eaae
Add build date udf mapping for fenix_nightly ( #1218 )
2020-08-06 15:32:45 -04:00
ac182ca56b
Add parsed_x_source_tags to metadata in user-facing views
...
As discussed in https://bugzilla.mozilla.org/show_bug.cgi?id=1657360#c1
2020-08-06 15:25:48 -04:00
c7eae14b25
Add submission_date to clients_scalar_aggregates ( #1216 )
2020-08-06 14:28:53 -04:00
d5d64359f6
Bug 1657360 Exclude pings with "automation" tag from stable
...
We will also need to update monitoring queries to account for this when
counting unique document_ids in decoded and live tables.
2020-08-06 12:56:15 -04:00
2acb30c9b0
Rewrite duplicated map udfs to mozfun shims ( #1211 )
...
* Rewrite duplicated map udfs to mozfun shims
* Format get_key_with_null.sql
2020-08-04 13:26:13 -07:00
8192ee323f
Fix URI comparison ( #1205 )
...
* Fix URI comparison
No need to backfill for this now. Just fixing for future use.
* Fix formatting.
Co-authored-by: Jeff Klukas <jklukas@mozilla.com>
2020-08-03 09:49:54 -04:00
0969a51673
Include sample_id in feature_usage ( #1201 )
...
Used as clustering key and allows sampling
2020-08-03 09:38:02 -04:00
cc06758a9a
Add tagged_search current to LTV ( #1210 )
2020-07-31 15:48:34 -04:00
8d6c2f47c1
Add sqrt to LTV calc ( #1202 )
...
* Add sqrt to LTV calc
* Format query
Co-authored-by: Frank Bertsch <fbertsch@mozilla.com>
2020-07-31 10:48:20 -04:00
019666b51b
Add queries for exported app store data ( #1207 )
2020-07-29 18:02:16 -04:00
a39e168a2d
Update query scheduling section
2020-07-29 14:11:00 -07:00
111382b37a
Rewrite duplicated bits28 udfs to mozfun shims ( #1208 )
2020-07-29 11:17:26 -07:00
6dedaca002
Improve query CLI based on feedback
2020-07-29 08:26:24 -07:00
8171e30010
Refactor CLI
2020-07-29 08:26:24 -07:00
bb47afedd5
Docs for query CLI command
2020-07-29 08:26:24 -07:00
fee4f44e9a
Fix CLI version
2020-07-29 08:26:24 -07:00
Anna Scholtz
Jeff Klukas
Anthony Miyaguchi
Ben Wu
William Lachance
Jesse McCrosky
asiOvOtus
Jeff Klukas
jmccrosky
Ben Miroglio