6b5ad9a55a
Publish views alternate project support
2020-10-05 12:59:58 -07:00
f6c67c25f2
Support alternate projects in publish_static
2020-10-05 12:59:58 -07:00
0a141f57bf
Project support for publishing UDFs
2020-10-05 12:59:58 -07:00
e010b0ec3c
Move files to moz-fx-data-shared-prod
2020-10-05 12:59:58 -07:00
074c3a44f8
Bump smart-open from 2.2.0 to 2.2.1 ( #1370 )
2020-10-05 19:32:39 +00:00
76a20ab949
Bump pytest from 6.1.0 to 6.1.1 ( #1381 )
2020-10-05 18:22:25 +00:00
bbd15f8ab5
Update ping selection criteria for fission monitoring tables
...
This adds filtering by experiment slug and filters out sessions where fission hasn't yet been enabled.
2020-10-05 18:59:19 +02:00
157209f2f5
Use submission_timestamp for partitioning in fission monitoring table ( #1372 )
2020-10-05 12:52:30 +02:00
bc5e4292fa
Update stripe schemas and fix subscription imports ( #1379 )
2020-10-02 12:48:43 -07:00
c6888b57e5
Update destination table for bigquery_etl_scheduled_query_usage
2020-10-02 12:25:38 -07:00
60bf1347fb
Added visited_* columns to aet_clients_daily ( #1375 )
...
Quick follow-up to #1371
2020-10-02 14:39:21 -04:00
11a24e6b5a
Refactor query usage monitoring query
2020-10-02 10:38:59 -07:00
6e456168b8
Add query usages monitoring
2020-10-02 10:38:59 -07:00
ad896eed82
Remove anomdtct
2020-10-02 10:31:42 -07:00
2d37c4f33f
Reformat search_clients_last_seen
2020-10-02 10:15:53 -07:00
06d78415ed
Remove `total_searches`
...
The way `total_searches` is defined as sum over several other search-related metrics is not proper (but ok in determining whether user searched as used in this case). Misusage of total_searches value can be misleading, so we prefer to remove it from the view.
2020-10-02 10:15:53 -07:00
abb2ae5d90
Bug fixes for AET daily ETL ( #1371 )
...
This fixes a bug about missing uids in the daily table as reported in
https://bugzilla.mozilla.org/show_bug.cgi?id=1637926#c5
It also adds some boolean fields to allow the daily table to specifically
address questions about whether a given client was 5-uri active per day.
2020-10-02 12:50:32 -04:00
fdad753b1c
Remove `total_searches` ( #1368 )
...
The waytotal_searches is defined as sum over several other search-related metrics is not proper (but ok in determining whether user searched as used in this case). Misusage of total_searches value can be misleading, so we prefer to remove it from the view.
Co-authored-by: Frank Bertsch <fbertsch@mozilla.com>
2020-10-01 20:53:11 -04:00
7d40d8cde0
Fix bqetl CLI setup
2020-10-01 16:21:07 -07:00
0ed408e7dd
Fix #1329 - Use app_build_id as app_version in GLAM fenix nightly ( #1354 )
...
* Make versions to keep configurable
* Replace app_version with app_build_id in nightly
* Add jsonschema as a requirement=
* Filter based on build date instead of version for nightly
* Add script for comparing the output of two branches
* Add option for specifying the bucket in export
* Cast build_id to integer
* Remove latest versions from histogram aggregates
* Format logical_app_id
* Use @submission_date parameter in latest versions
2020-10-01 14:28:42 -07:00
efb49cf738
Dryrun stored procedures ( #1367 )
...
* Dryrun stored procedures
* Reformat SQL
* Update tests for reformatted sql
* Fix doc format again
* Let black run past max-line-length
Black does not mandate line-length, it only tries to adhere to it.
For that reason, we can't have flake8 fail at too-long line length.
https://black.readthedocs.io/en/stable/the_black_code_style.html#line-length
* Why doesn't pydoctest run locally?
* Apply suggestions from code review
Co-authored-by: Anna Scholtz <anna@scholtzan.net>
Co-authored-by: Anna Scholtz <anna@scholtzan.net>
2020-10-01 16:48:28 -04:00
f8dc0cfa5e
Add tables for monitoring fission experiment ( #1327 )
2020-10-01 19:44:33 +02:00
b25bdbd415
Bump google-cloud-bigquery from 1.28.0 to 2.0.0 ( #1364 )
2020-10-01 16:43:00 +00:00
ceeaf302cc
Bump gitpython from 3.1.8 to 3.1.9 ( #1363 )
2020-10-01 16:34:57 +00:00
ffa0c7e0c8
Bump stripe from 2.53.0 to 2.54.0 ( #1362 )
...
Bumps [stripe](https://github.com/stripe/stripe-python ) from 2.53.0 to 2.54.0.
- [Release notes](https://github.com/stripe/stripe-python/releases )
- [Changelog](https://github.com/stripe/stripe-python/blob/master/CHANGELOG.md )
- [Commits](https://github.com/stripe/stripe-python/compare/v2.53.0...v2.54.0 )
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2020-10-01 09:26:20 -07:00
bcb95bb449
Allow documentation for stored procedures ( #1358 )
...
* Allow documentation for stored procedures
Add docs for `create_events_view` procedure
* Additional instruction on procedure usage
* Fix SQL; fix backticks
* Last backtick fix
* Apply suggestions from review
* Update procedure reference
* Fix other use of backticks
* Add rough schema outline
* Apply suggestions from code review
Co-authored-by: William Lachance <wrlach@gmail.com>
Co-authored-by: William Lachance <wrlach@gmail.com>
2020-09-30 15:53:23 -04:00
dbadafa790
Fix #1356 - Add extra validation before casting 8 digit build ids ( #1357 )
...
* Add extra validation before casting 8 digit build ids
* Move condition into main CASE statement
* Move length into case bodies
2020-09-30 10:54:30 -07:00
af37c47f5f
Add support for publishing procedures ( #1355 )
2020-09-29 20:05:45 -04:00
3caa5eb382
Event analysis udfs ( #1343 )
...
* WIP: Event analysis UDFs
* WIP: Some stored procedures for working with events
* Add UDF descriptions && move datasets
* Add capabilities for testing procedures
* Add procedures for creating funnel views
* Dryrun stored procedures
* Add support for count queries to events view
* Remove legacy stored procedures
* Remove dryruns
* Add explicit count support
* Change SQL to work around sqlparse bug
See https://github.com/andialbrecht/sqlparse/issues/583
* Run black
* Reformat SQL
* Reformat assertion
* Fix doc issues
* Add stored procedure test file
* Flake8 issue
* Reformat last file
* Ignore pre-colon whitespace failure
* Also ignore line breaks before binary operators
* Apply suggestions from code review
Co-authored-by: Anna Scholtz <anna@scholtzan.net>
* Fix sql lint
Co-authored-by: Anna Scholtz <anna@scholtzan.net>
2020-09-29 16:33:38 -04:00
695e443706
Bump smart-open from 2.1.1 to 2.2.0
...
Bumps [smart-open](https://github.com/piskvorky/smart_open ) from 2.1.1 to 2.2.0.
- [Release notes](https://github.com/piskvorky/smart_open/releases )
- [Changelog](https://github.com/RaRe-Technologies/smart_open/blob/develop/CHANGELOG.md )
- [Commits](https://github.com/piskvorky/smart_open/compare/2.1.1...2.2.0 )
Signed-off-by: dependabot[bot] <support@github.com>
2020-09-29 01:15:13 +00:00
83dc1a6ed7
Bump pytest from 6.0.2 to 6.1.0
...
Bumps [pytest](https://github.com/pytest-dev/pytest ) from 6.0.2 to 6.1.0.
- [Release notes](https://github.com/pytest-dev/pytest/releases )
- [Changelog](https://github.com/pytest-dev/pytest/blob/master/CHANGELOG.rst )
- [Commits](https://github.com/pytest-dev/pytest/compare/6.0.2...6.1.0 )
Signed-off-by: dependabot[bot] <support@github.com>
2020-09-29 00:34:06 +00:00
e40b780e65
Add Japan to country list for GUD ( #1352 )
...
See https://github.com/mozilla/gud/issues/112
2020-09-28 15:49:40 -04:00
0204d50e85
Simplify CLI UDF rename and allow wildcard patterns
2020-09-25 14:06:41 -07:00
cb0b5f5ba3
CLI UDF rename command
2020-09-25 14:06:41 -07:00
b8605f638f
UDF rename command
2020-09-25 14:06:41 -07:00
1e3546a4df
Disable destination_table for script.sql ( #1346 )
...
Bug fix for #1288
Currently, the one script.sql task we have fails in Airflow due to having
destination_table set in the query job, which is disallowed for scripts.
2020-09-25 14:39:22 -04:00
7a7777c198
Bump stripe from 2.51.0 to 2.53.0 ( #1345 )
2020-09-25 17:52:33 +00:00
9bcd1d889c
Bug 1637926 Add an aet_clients_daily table for end-user analysis ( #1288 )
2020-09-25 12:21:58 -04:00
b619159cfc
Rename QUERY_FILE_RE to SQL_FILE_RE
2020-09-25 08:59:38 -07:00
a557992b76
Use name patterns for UDF CLI commands
2020-09-25 08:59:38 -07:00
7c76228616
Update CLI query tests to use name patterns
2020-09-25 08:59:38 -07:00
d70ad34a92
Use name patterns for CLI commands
2020-09-25 08:59:38 -07:00
4dc32cede4
Support name patterns for query info and query scheduling
2020-09-25 08:59:38 -07:00
593c88a1f0
Bump google-cloud-storage from 1.31.1 to 1.31.2 ( #1339 )
2020-09-24 12:59:33 -07:00
effd1d3394
Fix path for bqetl udf cli ( #1340 )
...
* Fix path for bqetl udf cli
* Reformat file with black
2020-09-24 13:33:53 -04:00
3527ccca51
Refactoring
2020-09-24 08:55:35 -07:00
2e56471644
Move run_multipart_query logic to bigquery_etl
2020-09-24 08:55:35 -07:00
a604268c7e
Move publish_static to bigquery_etl
2020-09-24 08:55:35 -07:00
00a36c3553
Move json_to_table_ddl to bigquery_etl
2020-09-24 08:55:35 -07:00
08be8da2a1
Move generate_incremental_table logic to bigquery_etl
2020-09-24 08:55:35 -07:00
Anna Scholtz
dependabot[bot]
Arkadiusz Komarzewski
Arkadiusz Komarzewski
Daniel Thorn
Jeff Klukas
XuanL
Anthony Miyaguchi
Frank Bertsch