* glam: Partition clients_histogram_aggregates by sample_id (has been running like this since April 3 from a different branch)
* glam: Non normalized aggregations to legacy histograms
* glam: add non-normalized aggs to probe counts extract
* glam: add init.sql to relevant tbls for non-norm aggs
* glam: ignore dryrun histogram_percentiles
* glam: add description and eol to init
* glam: Partition clients_histogram_aggregates by sample_id (has been running like this since April 3 from a different branch)
* glam: Non normalized aggregations to legacy histograms
* glam: add non-normalized aggs to probe counts extract
* glam: add init.sql to relevant tbls for non-norm aggs
* glam: ignore dryrun histogram_percentiles
* glam: add description and eol to init
* fix schema files
* fix clients_histogram_probe_counts schema
* remove another init.sql
* fix dryrun ignore order
* fix table name
* change dryrun ignore order to try avoiding fenix for being on path
* another change in dryrun
* Move glam queries from dryrun to bqetl_project.yaml to ignore
* add tbl deps on tests
* Include new fields for SubPlat in `fxa_content_auth_stdout_events`.
* Include new fields for SubPlat in `nonprod_fxa_content_auth_stdout_events`.
* Include new fields for SubPlat in `fxa_all_events`.
* Move new `time` column to be by the other timestamp columns.
* Keep `subscribed_plan_ids` as a string so it's accessible in Looker.
* Add `schema.yaml` files for FxA events ETLs.
So the tables can be successfully staged for CI for downstream ETLs/views to pass.
* Fully qualify view in `fxa_users_daily_v1` to try to get test to pass.
* Rename `time` column `event_time`.
* Include new fields for SubPlat in `nonprod_fxa_all_events`.
---------
Co-authored-by: Daniel Thorn <dthorn@mozilla.com>
* DENG-850 Retrieve FALSE instead of NULL in the in metadata when there isn't first_session or metrics ping.
* DENG-850 Unitest for no first session ping.
* DENG-850 syntax fix
* DENG-850 Tests for first session ping, and no baseline ping.
* DENG-850 Tests suite.
* DENG-850 YAML fixes.
* DENG-850 Adjustment to the case of reported first_session and metrics ping. The unitests are adjusted to get the value for reported pings.
* DENG-850 Add sample id to test.
---------
Co-authored-by: Lucia Vargas <lvargas@mozilla.com>
* DENG-775 Added session_id to JOIN between GA data and stub_attr.stdout. Also expanded date range on GA session data to [download_date - 2 days, download_date + 1 day]
* Updated query to handle missing GA download_session_id. It effectively applies V1 logic to the MISSING_GA_CLIENT dl_tokens.
* Initial table definitions for dl_token processing. Includes update to sql pytest_plugin to account for tablenames with date suffixes.
* Removed cluster reference and shortened description
* Added sql/moz-fx-data-marketing-prod/ga_derived/downloads_with_attribution_v1/query.sql to dryrun skip
* Added time_on_site
* Moved country_names sample test data file.
* Update bigquery_etl/pytest_plugin/sql.py
Co-authored-by: Daniel Thorn <dthorn@mozilla.com>
* Update sql/moz-fx-data-marketing-prod/ga_derived/downloads_with_attribution_v1/query.sql
Co-authored-by: Frank Bertsch <frank.bertsch@gmail.com>
* Update sql/moz-fx-data-marketing-prod/ga_derived/downloads_with_attribution_v1/query.sql
Co-authored-by: Frank Bertsch <frank.bertsch@gmail.com>
* Updated based on PR feedback. Added LEFT JOIN to ensure sessions without pageviews are not dropped.
* Set has_ga_download_event = null if exception=GA_UNRESOLVABLE
* Standardized logic for time_on_site
* - Added test for multiple downloads for 1 session
- Added detailed description of table.
* Updated to use mode_last_retain_nulls instead of ANY_VALUE
* Set pageviews, unique_pageviews = 0 if null.
* Added boolean additional_download_occurred to indicate if another download occurred in the same session.
---------
Co-authored-by: Daniel Thorn <dthorn@mozilla.com>
Co-authored-by: Frank Bertsch <frank.bertsch@gmail.com>
* feat: new field in search clients daily - is_sap_monetizable
* Added column to tests
---------
Co-authored-by: Alexander Nicholson <anicholson@mozilla.com>
* Revert "CI fixes for supporting private UDFs in bigquery-etl - DENG-735 (#3631)"
This reverts commit edcfe758f7.
* Added stub UDF for monetized_search
* Add docs for using a private internal UDF
* Minimize stub normalize_search_engine UDF and usage in search_clients_last_seen tests
* Move sql tests downstream of private-generate-sql and copy UDFs into sql-dir for tests
* Add sidebar search probes to clients_daily tables
* update downstream schemas
* Format fix
* Add new field to test
* Update schemas with hist fields
* Remove duplicated field from schema
---------
Co-authored-by: Glenda Leonard <75265513+gleonard-m@users.noreply.github.com>
* Add urlbar_persisted to query for daily search client table
* Add column to schema (not backward breaking)
* Address test expectations
* Update schemata and queries for companion tables
* Make adjustments that Alex identified in the PR to make sure the new fields get ingested properly
* Run schema update for clients_daily_v6
* Update the suggest_impression_sanitized_v3 query
* exclude region and country when preparing for the join
* filter `impressions.request_id` to non null to drop queries without a
corresponding impression.
* Add tests? Haven't figured out bootstraping issues on my M1 yet so not
sure how well these will work. TO CI!
* Swap the left and right and remove the conditional on the final join
* Align expectations
* Will this fix the tests? tune in to find out.
* Fix expectations AGAIN
* Update based on review comments and formatter changes
This is a follow-up to https://github.com/mozilla/bigquery-etl/pull/3037 which unblocked `scalar_bucket_counts_v1`.
`scalar_percentiles_v1` uses the same source table (`clients_scalar_aggregates_v1`) and started failing today with the same error (disk/memory limits exceeded for shuffle operations).
`APPROX_COUNT_DISTINCT` used here runs HLL under the hood. The reason for using it here is that we can't split the aggregation here into two stages as in the aforementioned PR due to quantiles calculation.
I have run this query locally and confirmed that it works.
* CONSVC-1681 Add mobile data to contextual services event_aggregates
See https://mozilla-hub.atlassian.net/browse/CONSVC-1681
* Use 'phone' instead of 'mobile'
* Update init.sql
* Commentary on filter
* Aggregation test update
* Update overactive filter test
* Dry run exemptions
* Update sql/moz-fx-data-shared-prod/contextual_services_derived/event_aggregates_v1/query.sql
* format
This better matches the current client behavior for matching.
We're currently getting `<disallowed>` in results and results with uppercase
letters. I don't think preserving these differences has analytical value,
and it makes the results harder to work with.
* ROAD-85 Simple sanitization job for Merino logs
See https://mozilla-hub.atlassian.net/browse/ROAD-85
This uses the adM allowlist of queries for sanitization, so can be expressed
entirely in a single query. Future iterations will involve python logic and
will likely need to be held elsewhere.
* Separate external query and query to copy data into shared-prod
Winnie Chan
Eduardo Filho
Rebecca BurWei
Sean Rose
Alexander
Lucia
Glenda Leonard
Anna Scholtz
kik-kik
m-d-bowerman
Leli
Daniel Thorn
Curtis Morales
Chelsea Troy
Alekhya
Frank Bertsch
wil stuckey
Nan Jiang
wil stuckey
akkomar
Jeff Klukas