* Bug 1666768 Truncate histogram values above 2^31
Reverts the temporary fix from #1333
* Use LEAST
Co-authored-by: Daniel Thorn <dthorn@mozilla.com>
Co-authored-by: Daniel Thorn <dthorn@mozilla.com>
* Add glam cli for listing processed app ids
* Make backfill scripts more consistent
* Add export to glam glean cli
* Add pandas dependency
* Add black format of glam-cli
* Commit hashes based on bigquery-etl container
* Fix various linting issues
* Be stricter with is_logical matching
* Fix more linting issues
* Bug 1635906 Add bqetl support for scripts and script for AET lookup
There are some code changes here for DAG generation and for testing.
* Apply suggestions from code review
Co-authored-by: Anna Scholtz <anna@scholtzan.net>
* DAG fixups
Co-authored-by: Anna Scholtz <anna@scholtzan.net>
Microsoft will be rolling out the October 2020 update (version 20H2) soon. Update the query to break on the final build number for it, 19042.
Co-authored-by: Frank Bertsch <fbertsch@mozilla.com>
* Update query.sql
Hide revenue and total LTV fields from view
* Update sql/revenue_derived/client_ltv_normalized/query.sql
Co-authored-by: Frank Bertsch <fbertsch@mozilla.com>
This adds a new `environment` parameter to Shredder delete script, accepting either `telemetry` or `pioneer` values.
The goal of this parameter is to have a way for choosing different source and target tables. By default it is set to `telemetry` (which keeps current behavior unchanged).
Next step is to define logic for discovering Pioneer study tables in `config.py` and invoke it in delete script (`NotImplementedError`).
* Add option to specify (or not) dest table
* Add option to DAG generation for single dag
* Add query for recent day's event_types
* Use explicit None for no destination table
* Reformat SQL
* Use event_types view in events_daily
* Update tests
* Run black
* Add view for events_daily
* Format SQL
* Ignore event properties in query
* Shorten events_name_v1
* Fix naming for test
* Update README.md
Co-authored-by: Anna Scholtz <anna@scholtzan.net>
Co-authored-by: Anna Scholtz <anna@scholtzan.net>
* Create DAG for events rollup
* Update sql/org_mozilla_firefox_derived/events_daily_v1/metadata.yaml
Co-authored-by: Anna Scholtz <anna@scholtzan.net>
Co-authored-by: Anna Scholtz <anna@scholtzan.net>
* Create events daily rollup
This takes the most recent days' data, rolls up the events,
and encodes them as unicode.
Add tests for android events daily
* Remove unecessary file
* Use map.mode_last
* Reformat sql
* Fix experiment aggregation
* Address review feedback
* Fix submission_date
* Git proper alias to events table
* Query for update event_types_v1
This query takes yesterday's events, yesterday's event_types,
and adds the new events, event_properties, and property values.
It writes it out to a new partition. This is not strictly
necessary but will aid debugging and redoes.
* Format SQL
* Init SQL for event_types_v1
* Fix comparison of differently-sized lists
* Add support for tests of init stmts
* Include metadata for event_types_v1
* Add tests for event_types init
* Reformat SQL
* Run black
* Skip invalid unicode sections in event_code_points_to_string
* Allow for init-only queries
* Partition events_daily_v1 by submission-date
This is not strictly required, but will aid in
debugging and reruns.
* Add assertion for not null
* Lint
* Alias events ping name
* Ignore time_ms event property
Anna Scholtz
Jeff Klukas
Daniel Thorn
Anthony Miyaguchi
dependabot[bot]
Ryan VanderMeulen
Frank Bertsch
Ben Miroglio
Arkadiusz Komarzewski
Jeff Klukas