* Replace GLAM temp functions with persistent functions
* Add generated sql
* Fix typo in udf name
* Add missing files and fully qualify udfs
* Add missing namespace
* Namespace even more things
* format sql
* Add bits_from_offsets UDF
This is relevant to the emerging "clients_all_time" work drafted in
https://github.com/mozilla/bigquery-etl/pull/1480
I'm proposing we add this under `udf` rather than in `mozfun` because I'm not
yet certain about the naming. If we want to have additional functionality to
support all-time bit patterns, I would like to have those organized under a
single `mozfun` namespace, and it's not clear yet what the interface should
look like.
This function on its own should be enough to empower a new DS workflow for
experimenting with new usage definitions before committing them to clients_daily
and clients_last_seen (to be documented).
* Resolve generated sql to glam-fenix-dev and change output in sql/ dir
* Add new script for testing glam-fenix queries
* Add generated sql for version control
* Use variables correctly in bash
* Remove latest versions from UDF
* Update test to generate minimum set of tables for nightly
* Commit generated queries for testing
* Cast only if not glob
* Ignore dryrun and publish view for glam-fenix-dev
* Fix linting error
* Update comments
* Use DST_PROJECT consistently in scripts
* Update comments
* Update script/glam/test/test_glean_org_mozilla_fenix_glam_nightly
Co-authored-by: Ben Wu <benjaminwu124@gmail.com>
* Update script/glam/generate_and_run_desktop_sql
Co-authored-by: Ben Wu <benjaminwu124@gmail.com>
Co-authored-by: Ben Wu <benjaminwu124@gmail.com>
* Exempt a few files from dry run due to new table-level ACLs
The dry run service can no longer perform queries with wildcard table
specifications or access raw AET data. See https://github.com/mozilla-services/cloudops-infra/pull/2599
* Verbose referenced_tables for AET logging clients daily
The `nondesktop_clients_last_seen_v1` view was developed mostly as an
internal implementation detail for downstream tables, but it has become
useful in its own right. This PR formalizes the view by providing an alias
without a version modifier and it adds a `product` field with application
names that are short but more meaningful than the `app_name` field.
See discussion in https://jira.mozilla.com/browse/DO-330 about confusion that
has resulted from the name "Fennec iOS" used in dashboards, etc. This is a
step toward reducing that kind of confusion.
This PR also adds `contributes_to_2019_kpi` and `contributes_to_2020_kpi` fields
as source of truth for how we count KPI metrics. That logic is currently
copied and pasted in several places, which could lead to errors.
This will need a fair amount of review from data users before moving forward.
It will also require backfilling several downstream tables and communicating
the change.
* Add initial incremental query for geckoview build dates
* Add initial tests for incremental query (WIP)
* Add files for initial tests
* Rework query so it doesn't fail during tests
* Fix schema so queries run
* Add passing test for init
* Add test for query aggregation
* Add metadata file for scheduling the query
* Move scripts from fenix_nightly to fenix
* Remove scheduling
* Add document strings.
* Change dataset reference and indent comments correctly
* Remove init and address feedback
* remove init file
* make query idempotent by appending window to each submission_date
* rename n_builds to n_pings
* reduce window size from 30 days to 14 days
* avoid use of subqueries
* Update tests for query
* Fix tests
* Add failing test for 100
* Fix query so it work across fx100 boundary
* Add linting fixes
While looking at Shredder logs, I noticed that all entries for FxA-related
derived tables show 0 deletions, like:
> 712215424909 bytes and deleted 0 rows from moz-fx-data-shared-prod.firefox_accounts_derived.fxa_users_daily_v1
It appears that `account.deleted` events populate a field named `uid` rather
than `user_id`. I was able to verify this by choosing a recent `uid` value
from a deletion event and counting events where that same value appears as
`user_id` in FxA logs. There were matching messages.
- Scrape all projects for routine defns when generating tests
- Create UDFs as non-temp for stored procedure tests
- Make assert functions default non-temp (to support above)
* Bug 1669516 Use `app_display_version` for Fenix AMO stats
* Use geckoview_version for fenix nightly
* Remove deprecated installs_v1
* Remove dev installs_v1
Anthony Miyaguchi
Jeff Klukas
Daniel Thorn
Sunah Suh
Rhys
Ben Wu
William Lachance
jailang
Anna Scholtz
Frank Bertsch