* Create DAG for events rollup
* Update sql/org_mozilla_firefox_derived/events_daily_v1/metadata.yaml
Co-authored-by: Anna Scholtz <anna@scholtzan.net>
Co-authored-by: Anna Scholtz <anna@scholtzan.net>
* Create events daily rollup
This takes the most recent days' data, rolls up the events,
and encodes them as unicode.
Add tests for android events daily
* Remove unecessary file
* Use map.mode_last
* Reformat sql
* Fix experiment aggregation
* Address review feedback
* Fix submission_date
* Git proper alias to events table
* Query for update event_types_v1
This query takes yesterday's events, yesterday's event_types,
and adds the new events, event_properties, and property values.
It writes it out to a new partition. This is not strictly
necessary but will aid debugging and redoes.
* Format SQL
* Init SQL for event_types_v1
* Fix comparison of differently-sized lists
* Add support for tests of init stmts
* Include metadata for event_types_v1
* Add tests for event_types init
* Reformat SQL
* Run black
* Skip invalid unicode sections in event_code_points_to_string
* Allow for init-only queries
* Partition events_daily_v1 by submission-date
This is not strictly required, but will aid in
debugging and reruns.
* Add assertion for not null
* Lint
* Alias events ping name
* Ignore time_ms event property
Airflow's publishing of mozfun definitions failed last night due to
https://github.com/mozilla/bigquery-etl/pull/1287
Long-term we should enforce that mozfun function definitions don't directly
reference mozfun.
See https://github.com/mozilla/bigquery-etl/pull/1250 for proposed logic of how
to parse a date from the 10-digit format, but this change should be reliable
for the specific case of determining whether we are before or after the
2020-07-03 epoch where naming of apps and channels changed.
This will affect GLAM and GUD.
Reverts https://github.com/mozilla/bigquery-etl/pull/1249
I will plan to run a backfill for GUD on the same day I merge this, and then
communicate out the change to data users, since channel breakdowns in GUD
will change for Fenix and Firefox Preview.
* Add `clients_last_seen.days_any_interaction_bits`.
This column is a 28-day bit array containing whether the client has any active ticks (thus nonzero active hours) associated with each of the last 28 submission dates.
* Change column name.
Follow the grammatical pattern for these column names, as pointed out by @jklukas.
* Update tests
* Reconcile order to satisfy IF
Co-authored-by: Jeff Klukas <jklukas@mozilla.com>
* Fix writing to temporary table when publishing JSON
* Update test_script_incremental_query test
* Ensure temporary artifacts are deleted
* Wrap deletion of artifacts into try finally block
Frank Bertsch
Jeff Klukas
Anna Scholtz
Ben Miroglio
Jeff Klukas
dependabot[bot]
Felix Lawrence
Daniel Thorn