Граф коммитов

3225 Коммитов

Автор SHA1 Сообщение Дата
kik-kik 055a5d96a6
pushing bqetl acoustic dags by 1 hour to see if that resolves sensor issues (#3002) 2022-06-02 15:18:41 +02:00
perrymcmanis144 4e5684416f
Decision support metrics for fog initial commit (#2984)
* decision support metrics for fog initial commit

* reverting overwrite

* init.sql to initialize table

* changing to main_1pct for the time being

* init sql correction

* missing as keyword

* updating dag order

* trying main 1pct

* removed init.sql, added schema.yaml

* update dag to reflect different table wait

* re-generated DAG

* fix tag

* fix tags in dag

* no clustering
2022-06-01 16:00:29 -05:00
Jeff Klukas c7049d8d52
Bug 1770814 adm_daily_aggregates (#2999)
* Bug 1770814 adm_daily_aggregates

See https://bugzilla.mozilla.org/show_bug.cgi?id=1770814

* Remove filter on aggregated_search_Terms_daily_v1
2022-06-01 20:15:34 +00:00
akkomar 2b7d6ddeca
DSRE-775 Add support for Rally Web Platform unenrollment ping in Shredder (#3001)
This excludes `unenrollment_v1` table from deletion process - similar to other enrollment/unenrollment tables we don't want to shred it.
2022-06-01 19:59:19 +00:00
Jeff Klukas 00db2b884d
Revert "Add Focus, Firefox iOS, Klar to `nondesktop_clients_last_seen` (#2998)" (#3003)
This reverts commit 306e5ea9b6.
2022-06-01 17:34:45 +00:00
kik-kik de2c7ca1dd
removed link to Google Docs as public should not try to access it (#3000) 2022-06-01 14:45:37 +00:00
Alessio Placitelli 306e5ea9b6
Add Focus, Firefox iOS, Klar to `nondesktop_clients_last_seen` (#2998)
These products were not added as their respective clients_last_seen
tables, which are powered by Glean data, were not genrated.
We need them in this view so that Focus/Firefox iOS/Klar can
track their KPI using Glean data as well.
2022-06-01 07:11:37 -07:00
Daniel Thorn cb0b9cbee4
Bug 1768419 - Add note to subplat dag docs for airflow triage (#2959)
* Bug 1768419 - Add note to subplat dag docs for airflow triage

* enriched the bqetl_subplat DAG description with a bit more detail regarding expected failures

Co-authored-by: kik-kik <kignasiak01@gmail.com>
2022-06-01 15:54:19 +02:00
Sean Rose 4ed36cca3e
Account for coupons for production CJMS subscriptions. (#2995) 2022-05-31 19:33:01 +00:00
dependabot[bot] fbb2ab30d7
Bump pathos from 0.2.8 to 0.2.9 (#2993) 2022-05-31 16:38:25 +00:00
dependabot[bot] 14c39bf366
Bump mkdocs-material from 8.2.15 to 8.2.16 (#2996) 2022-05-31 16:21:53 +00:00
kik-kik 4e5694ad54
fixing parameter passing for the bqetl process (#2997) 2022-05-30 18:47:32 +02:00
Lucia 77c634e634
Aggregation for Attribution details (#2991)
* Aggregation for attribution analysis.

* Clustering and metadata. Add field attributed for easy filtering.

* Update new_profiles in query, add attributed field to schema

* Metadata details added.

Co-authored-by: Lucia Vargas <lvargas@mozilla.com>
2022-05-30 09:28:57 +02:00
Sean Rose 294a9f0d14
Account for coupons for non-prod CJMS subscriptions. (#2994) 2022-05-26 14:03:01 -07:00
dependabot[bot] 31db89ac83
Bump stripe from 3.1.0 to 3.2.0 (#2988) 2022-05-24 21:31:14 +00:00
dependabot[bot] 042f02079e
Bump gcsfs from 2022.3.0 to 2022.5.0 (#2985) 2022-05-24 21:14:09 +00:00
dependabot[bot] 1fec542726
Bump pip-tools from 6.6.1 to 6.6.2 (#2989) 2022-05-24 20:58:05 +00:00
Lucia 5cd23c0c96
Use static dataset to convert locale into the corresponding ISO 639-1… (#2987)
* Remove first_seen_date from the query to reduce the agg size

* Change first_seen_date for first_seen_year

* Change first_seen_date for first_seen_year

* Change first_seen_date for first_seen_year

* Reformat query

* Schema update

* Update query and schema.yaml for active_users_aggregates.

* Schema update

* Use static dataset to convert locale into the corresponding ISO 639-1 language name.

* Update query to get the locale code in the subquery

* Aggregation for device and os analysis

* Add aggregation to analyze device and os.

* Merge remote branch and update metadata

* Update calculation of new_profiles

* Schema update

* Update new_profiles calculation

* Update DAG and schema files

* Remove locale field, to be replaced for language_name

Co-authored-by: Lucia Vargas <lvargas@mozilla.com>
2022-05-24 19:48:15 +02:00
Jeff Klukas ec3059c0ee
Bug 1757216 Mark BrowserStack pings in unified_metrics (#2936)
See https://bugzilla.mozilla.org/show_bug.cgi?id=1757216
2022-05-24 17:30:07 +00:00
Lucia 66e14ce615
Aggregation for OS and Device (#2992)
* Remove first_seen_date from the query to reduce the agg size

* Change first_seen_date for first_seen_year

* Change first_seen_date for first_seen_year

* Change first_seen_date for first_seen_year

* Reformat query

* Schema update

* Add aggregation to analyze device and os.

* Merge remote branch and update metadata

* Update new_profiles calculation

* More descriptive metadata

Co-authored-by: Lucia Vargas <lvargas@mozilla.com>
2022-05-24 19:09:19 +02:00
Lucia c9532788c3
Separate device model and attribution granularity from active_users_aggregates (#2986)
* Update query and schema.yaml for active_users_aggregates.

* Schema update

* Update calculation of new_profiles

* Update new_profiles calculation

Co-authored-by: Lucia Vargas <lvargas@mozilla.com>
2022-05-24 17:31:42 +02:00
Lucia 6dd08ae652
Calculation of new_profiles in active_users_aggregates (#2980)
* Remove first_seen_date from the query to reduce the agg size

* Change first_seen_date for first_seen_year

* Change first_seen_date for first_seen_year

* Change first_seen_date for first_seen_year

* Reformat query

* Schema update

* Update calculation of new_profiles

* Query file reformatted

Co-authored-by: Lucia Vargas <lvargas@mozilla.com>
2022-05-24 16:13:25 +02:00
Lucia c3e8250c2f
Correct the join between clients and searches data for mobile. (#2956)
* Correct the join between clients and searches data for mobile.

* Add days_created_profile_bits, query only required fields for unified metrics: remove and extend query.

* Handle country NULLS to join with country lookup.

* Update schema.yaml

* Update query to fit schema.yaml

Co-authored-by: Lucia Vargas <lvargas@mozilla.com>
2022-05-23 11:50:29 +02:00
Lucia dfef0030cf
Update docs to remove the default mozilla-public-data as default in docs (#2960)
* Update docs to remove the default instruction to use  `mozilla-public-data` as the project to create a new table. Clarify to specify the project.

* Update documentation to create a derived dataset. Replace the use of init files for the use of schemas.

* Small corrections

* Small corrections

Co-authored-by: Lucia Vargas <lvargas@mozilla.com>
2022-05-23 11:23:17 +02:00
dependabot[bot] 64e65ef019
Bump stripe from 3.0.0 to 3.1.0 (#2983) 2022-05-20 18:36:37 +00:00
Lucia cdf2357546
Calculate is_new_profile using first_seen_date for Fenix, Focus and Firefox iOS (#2977)
Co-authored-by: Lucia Vargas <lvargas@mozilla.com>
2022-05-20 12:58:58 +02:00
Lucia 6df26bba60
Change first_seen_date for first_seen_year in active_users_aggregates (#2978)
* Remove first_seen_date from the query to reduce the agg size

* Change first_seen_date for first_seen_year

* Change first_seen_date for first_seen_year

* Change first_seen_date for first_seen_year

* Reformat query

* Schema update

Co-authored-by: Lucia Vargas <lvargas@mozilla.com>
2022-05-20 12:37:16 +02:00
Anna Scholtz 69c9a50e41 Fix experiment status 2022-05-19 12:36:28 -07:00
Anna Scholtz d01934c808 Fix experiments status 2022-05-19 12:36:28 -07:00
dependabot[bot] e87e2480c3
Bump ujson from 5.2.0 to 5.3.0 (#2981) 2022-05-19 11:48:51 -07:00
Jeff Klukas 54871b8724
Bug 1768507 Add nondistinct count to distinct_docid query (#2958)
See https://bugzilla.mozilla.org/show_bug.cgi?id=1768507

This is an alternate approach that replaces https://github.com/mozilla/bigquery-etl/pull/2953
and avoids adding a new table. This also gives us the benefit of allowing
easy access to duplicate rate in decoded tables.
2022-05-18 18:39:54 +00:00
Daniel Thorn 77a08f1785
Bug 1767487 - mitigate conflicting writes between shredder and ETL (#2975) 2022-05-17 16:10:14 +00:00
Alexander Nicholson 6b55c9e4f5
Shift retention one day back (#2974)
Shift retention analysis 1 day one day back
2022-05-17 11:33:54 -04:00
Daniel Thorn 5ea2f1dea2
Remove redundant GOOGLE_APPLICATION_CREDENTIALS in integration (#2973) 2022-05-16 19:53:50 +00:00
dependabot[bot] 6fa34a64c9
Bump pip-tools from 6.6.0 to 6.6.1 (#2966) 2022-05-16 19:35:52 +00:00
Alexander Nicholson d285782b5e
Use the query metadata's defined date_partition_parameter for CLI backfills (#2972) 2022-05-16 15:05:31 -04:00
Alexander Nicholson b4e27f90c2
Change partition strategy for cohort_daily_statistics (#2971) 2022-05-16 14:34:30 -04:00
dependabot[bot] 6c436041c3
Bump mkdocs-material from 8.2.14 to 8.2.15 (#2967) 2022-05-16 16:52:37 +00:00
Alexander Nicholson 361170cb20
Updated metadata and DAG for cohort_daily_statistics (#2970) 2022-05-16 12:35:53 -04:00
Alexander Nicholson e962f93ef3
Update query parameter for rolling_cohorts to match default passed parameter (#2969) 2022-05-16 11:37:32 -04:00
Alexander Nicholson 5f34aab153
Added bigquery.time_partitioning to metadata (#2968) 2022-05-16 11:09:13 -04:00
Alexander Nicholson 6122d01094
Cohort Daily Statistics table (#2965)
Create Cohort Daily Statistics table
2022-05-13 16:25:48 -04:00
Alexander Nicholson c865ce9ba9
Rolling cohorts table (#2962)
Added table tracking rolling cohorts.
2022-05-13 13:47:50 -04:00
Alexander Nicholson c5ddf186b2
Ignore leading 0s in extract_version and browser_version UDFs (#2963)
Ignore leading 0s in extract_version and browser_version UDFs
2022-05-13 10:14:56 -04:00
Lucia 76836ca7d9
Deng 72 add first seen date to active users (#2964)
* DENG-72 Update schema for active_users_aggregates, add first_seen_date

* Small syntax update on query

Co-authored-by: Lucia Vargas <lvargas@mozilla.com>
2022-05-13 12:08:41 +02:00
Lucia 20bd0301e8
Create derived dataset and DAG for the aggregation of active users. (#2952)
* Create derived dataset and DAG for the aggregation of active users.

* Change APROX_DISTINCT_COUNT for COUNT(DISTINCT)

* Remove join with country lookup to avoid dependencies. Added search measures. Reorder fields for clarity.

* Improve descriptions

* Add clustering by channel

* DAG update

* Clustering based on users most common filtering

* Add notification for analytics DAG

* agg_active_users to query from telemetry_derived.

* Add query to create agg_active_users

* Update sql/moz-fx-data-shared-prod/telemetry_derived/agg_active_users_v1/init.sql

Co-authored-by: kik-kik <42538694+kik-kik@users.noreply.github.com>

* Add date to init.sql

* Update sql/moz-fx-data-shared-prod/telemetry_derived/agg_active_users_v1/init.sql

Co-authored-by: kik-kik <42538694+kik-kik@users.noreply.github.com>

* Update sql/mozfun/bytes/zero_right/metadata.yaml

Co-authored-by: kik-kik <42538694+kik-kik@users.noreply.github.com>

* Modify metadata.yaml for agg_active_users

* Update sql/moz-fx-data-shared-prod/telemetry_derived/agg_active_users_v1/query.sql

Co-authored-by: kik-kik <42538694+kik-kik@users.noreply.github.com>

* Modify metadata.yaml for agg_active_users

* Remove CAST from init.sql for agg_active_users

* Update sql/moz-fx-data-shared-prod/telemetry_derived/agg_active_users_v1/init.sql

* Update project in ini.sql to `moz-fx-data-shared-prod`.

* Update name of aggregation and DAGS for consistency

* Add active_users_aggregates_v1 to align with current naming convention

* Update to active_users_aggregates

* Update query for active_users_aggregates

* Add uri_count and active_hours query for active_users_aggregates

* Format query, update DAG to remove agg_active_users_v1

* Generate DAG to correct CI error.

Co-authored-by: Lucia Vargas <lvargas@mozilla.com>
Co-authored-by: kik-kik <42538694+kik-kik@users.noreply.github.com>
2022-05-12 16:23:50 +02:00
Daniel Thorn f1648b842d
Unschedule import of vpn waitlist table that was removed upstream (#2961) 2022-05-11 19:43:38 +00:00
akkomar a72ecf7f59
GLAM ETL: filter out unofficial desktop builds in FOG pipeline (#2957)
GLAM ETL: filter out unofficial desktop builds in FOG pipeline

In https://github.com/mozilla/glam/issues/1941 it was discovered that some desktop clients send version "1024" in FOG telemetry. This is breaking GLAM aggregations because they focus on latest released versions. In order to address that, from now on we'll:
1. filter pings by build_id against official Mozilla builds published in Buildhub. This is equivalent to the filtering we already do in the legacy telemetry GLAM ETL: 9bca48821a/sql/moz-fx-data-shared-prod/telemetry_derived/clients_daily_keyed_scalar_aggregates_v1/query.sql (L3-L8)
2. Explicitly filter out pings with version "1024.0.0". At the moment we do not know why this version number is sent by clients (see https://bugzilla.mozilla.org/show_bug.cgi?id=1768187 for details)
2022-05-11 00:33:35 +02:00
dependabot[bot] 3e95a0f200
Bump mkdocs-material from 8.2.13 to 8.2.14 (#2951) 2022-05-09 23:01:36 +00:00
dependabot[bot] 88a3625f6a
Bump pyarrow from 7.0.0 to 8.0.0 (#2950) 2022-05-09 22:45:28 +00:00