* Verify the format of the original SQL, not the generated SQL.
The generated SQL gets reformatted by `bqetl query render`.
* Format all SQL.
* Quote column names containing Jinja expressions to prevent `bqetl format` causing invalid SQL.
* Adjust indentation of some comments to align with the formatted SQL.
* Refactor final `SELECT` in `telemetry_derived.clients_first_seen_v2` to work better with `bqetl format` SQL formatting.
* Fix trailing line comments breaking inline block formatting.
* Fix leading whitespace before Jinja comments not being preserved.
* Add `schema.yaml` for `firefox_ios_derived.baseline_clients_yearly_v1`.
So the `deploy-changes-to-stage` CI can work for the downstream `firefox_ios.baseline_clients_yearly` view.
* Add `schema.yaml` for `firefox_accounts_derived/fxa_users_services_daily_v1`.
So the `dry-run-sql` CI can work for the downstream `firefox_accounts_derived.fxa_users_services_last_seen_v1` ETL.
* Correct `schema.yaml` and `init.sql` for `firefox_accounts_derived.fxa_users_last_seen_v1`.
So the `dry-run-sql` CI can work for the downstream `firefox_accounts_derived.fxa_users_last_seen_v1` ETL.
* Fully qualify table reference in `init.sql` for `firefox_accounts_derived.fxa_users_last_seen_v1`.
So the table dependency will get detected by the `deploy-changes-to-stage` CI to deploy it so the `dry-run-sql` CI can work for the `init.sql` file.
* Improve `JinjaComment` inheritance and docstring.
* Implement `Line.ends_with_line_comment` property and refactor `inline_block_format()`.
* WWW Site Metrics Summary V2 - update query logic
* Update sql/moz-fx-data-marketing-prod/ga_derived/www_site_metrics_summary_v2/query.sql
Co-authored-by: Sean Rose <1994030+sean-rose@users.noreply.github.com>
---------
Co-authored-by: Sean Rose <1994030+sean-rose@users.noreply.github.com>
* Add diff_query_schema function for utils
* Use mozdata instead of shared-prod
Co-authored-by: Anna Scholtz <anna@scholtzan.net>
* Delete tables as cleanup
* Apply suggestions from code review
Co-authored-by: Sean Rose <1994030+sean-rose@users.noreply.github.com>
* Compare types, add differs col
* Reformat query
* Update sql/mozfun/utils/diff_query_schemas/README.md
Co-authored-by: Sean Rose <1994030+sean-rose@users.noreply.github.com>
---------
Co-authored-by: Anna Scholtz <anna@scholtzan.net>
Co-authored-by: Sean Rose <1994030+sean-rose@users.noreply.github.com>
* added existing fxa tables to shredder config
* Apply suggestions from code review
Co-authored-by: Sean Rose <1994030+sean-rose@users.noreply.github.com>
* removing some of the fxa tables from the config as suggested by srose
---------
Co-authored-by: Sean Rose <1994030+sean-rose@users.noreply.github.com>
* Descheduled all fxa queries using the old AWS based tables
* Apply suggestions from code review
Co-authored-by: Sean Rose <1994030+sean-rose@users.noreply.github.com>
* added scheduling back to fxa_stdout_events_v1 and the corresponding nonprod as they are still receiving new data
* Apply suggestions from code review
Co-authored-by: Sean Rose <1994030+sean-rose@users.noreply.github.com>
---------
Co-authored-by: Sean Rose <1994030+sean-rose@users.noreply.github.com>
* Initial commit - work in progress
* DENG-2262 - updating column names
* DENG-2262 - renaming columns and adding 2 new columns
* DENG-2262 - fix typo in data type in schema.yaml
* Add a generator for events stream tables
Open questions:
* How does init work?
* Is this manually triggered? How do we backfill to a certain date?
* Schema is defined in SQL query. How does this behave on changes in
the future?
* Configuration: Right now inline in Python. Should we change this?
TODO:
* check table Schema
* Store category and name separately to help with filtering and clustering
* Concat into full event name using array to avoid NULL issues
* events stream: Read allowed apps from project configuration
* event stream: Cluster by event category
* Remove trailing commas
Co-authored-by: Anna Scholtz <anna@scholtzan.net>
* Update sql_generators/glean_usage/events_stream.py
* Update sql_generators/glean_usage/templates/events_stream_v1.metadata.yaml
* Update sql_generators/glean_usage/templates/events_stream_v1.query.sql
* Update sql_generators/glean_usage/templates/events_stream_v1.query.sql
* Update sql_generators/glean_usage/templates/events_stream_v1.query.sql
* Update sql_generators/glean_usage/templates/events_stream_v1.query.sql
* Update sql_generators/glean_usage/templates/events_stream_v1.query.sql
* Update sql_generators/glean_usage/common.py
* Update sql_generators/glean_usage/events_stream.py
---------
Co-authored-by: Anna Scholtz <anna@scholtzan.net>
The history is ordered from first introduction of the metric to the
latest change.
So the last element is the _current_ valid value for expires (and types).
For example this can be seen for `top_sites.contile_reporting_url` in
Firefox Android which initially was set to expire in v112, but has been
change to never expire later. The current database still lists 112 for
it though.
Co-authored-by: Leli <33942105+lelilia@users.noreply.github.com>
Sean Rose
Alexander
kik-kik
Katie Windau
Winnie Chan
Frank Bertsch
whd
Marlene Hirose
Anna Scholtz
Jan-Erik Rediger
akkomar
Leli
Eduardo Filho
Lucia
dependabot[bot]