3885e889f1
No logic changes required, added a note to the views to for clarification ( #4942 )
2024-02-01 16:22:09 +01:00
ad156158c3
Update bounces col description on www_site_hits_v2 and update bounces… ( #4937 )
...
* Update bounces col description on www_site_hits_v2 and update bounces aggregation in site landing page metrics v2
* DENG-2492 - fix formatting of updated SQL
2024-02-01 08:43:18 -06:00
b3bb61cbbc
Add workgroup_access to GLAM prod tables ( #4936 )
2024-01-31 19:11:37 -05:00
ddae31eb34
Don't run private-generate-sql for each PR ( #4938 )
2024-01-31 15:24:53 -08:00
09bc346dec
DENG-2492 - create new www_site_landing_page_metrics_v2 ( #4933 )
...
* DENG-2492 - create new www_site_landing_page_metrics_v2
* DENG-2492 - update query and schema for www_site_landing_page_metrics_v2
* DENG-2492 - add more column descriptions
* DENG-2492 - add better col desc
2024-01-31 15:55:32 -06:00
b0a1a32246
Speed up generate-sql ( #4921 )
...
* Speed up glean_usage generator
* Refactoring
2024-01-31 12:08:29 -08:00
91069ea6d2
DENG-2492 - update logic for www_site_events_metrics_v2 ( #4930 )
...
* DENG-2492 - update logic for www_site_events_metrics_v2
* DENG-2492 - updating logic for www_site_events_metrics_v2
2024-01-31 11:35:17 -06:00
56a12c7001
events stream: Expose additional metrics as a new JSON column ( #4926 )
2024-01-31 17:51:12 +01:00
c1adbd3c32
DENG-2492 - filter out null session IDs ( #4928 )
...
* DENG-2492 - filter out null session IDs
* DENG-2492 - filter out null sessions
2024-01-31 10:31:56 -06:00
d019d4ebc0
AD-178 Add query_type to event_aggregates_suggest and event_aggregates ( #4927 )
...
* Add query_type to event_aggregates_suggest
* Format
* Add query_type to event_aggregates table
* Update event_aggregates tests
2024-01-31 11:27:12 -05:00
5c905e73c8
Revert "AD-178 quick_suggest query type field" ( #4925 )
2024-01-30 20:07:20 -05:00
451310a033
Create unique `gcp.json` temp files to avoid file contention from parallel tests. ( #4924 )
2024-01-30 16:03:16 -08:00
8faf087a7a
Revert "Revert "point monitoring.bigquery_usage view from monitoring_derived.bigquery…"" ( #4923 )
2024-01-30 15:26:52 -08:00
ebe4239c1c
DENG-2492 Improve metadata for new table www_site_events_metrics_v2 ( #4920 )
...
* DENG-2492 - update metadata yaml file with a better description
* DENG-2492 - improving column descriptions and metadata table desc
2024-01-30 16:38:52 -06:00
0da78c06aa
AD-178 quick_suggest query type field ( #4901 )
...
* (WIP) Add query type to suggest event aggregates
* Format
* Pull query_type from AMP suggest blocks
2024-01-30 17:23:12 -05:00
aef070684f
DENG-2492 - update row number logic ( #4919 )
...
* DENG-2492 - update row number logic
* DENG-2492 - update row number logic
* Fix formatting
2024-01-30 15:47:56 -06:00
39a89a95ea
Update clients_last_seen_v1 and clients_first_seen to clients_first_seen_v2 ( #4915 )
...
* Update to clients_first_seen_v1
* Update tests
2024-01-30 16:33:01 -05:00
86c0eca325
Update `bqetl format` to not have an extra blank line after Jinja expressions ( #4899 )
...
* Update `bqetl format` to not have an extra blank line after Jinja expressions.
* Update `bqetl format` to add a blank line before `#fail` and `#warn` comments.
2024-01-30 09:35:12 -08:00
a5d54ff598
DENG-2492 initial commit for fixing exit data ( #4914 )
...
* DENG-2492 initial commit for fixing exit data
* Update sql/moz-fx-data-marketing-prod/ga_derived/www_site_hits_v2/query.sql
Co-authored-by: Frank Bertsch <frank.bertsch@gmail.com>
---------
Co-authored-by: Frank Bertsch <frank.bertsch@gmail.com>
2024-01-30 11:11:59 -06:00
0c0bb632dd
Bump aiohttp from 3.9.0 to 3.9.2 ( #4913 )
...
Bumps [aiohttp](https://github.com/aio-libs/aiohttp ) from 3.9.0 to 3.9.2.
- [Release notes](https://github.com/aio-libs/aiohttp/releases )
- [Changelog](https://github.com/aio-libs/aiohttp/blob/master/CHANGES.rst )
- [Commits](https://github.com/aio-libs/aiohttp/compare/v3.9.0...v3.9.2 )
---
updated-dependencies:
- dependency-name: aiohttp
dependency-type: indirect
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-01-30 08:33:47 -08:00
ba81121dad
remove unwanted space in bqetl checks labels resulting in potentially not being picked up correctly ( #4908 )
2024-01-30 07:40:07 -05:00
5c887867e3
DENG-2492 - create new table www_site_events_metrics_v2 ( #4911 )
...
* DENG-2492 - initial commit for new table www_site_events_metrics_v2
* DENG-2492 - updating query
* DENG-2492 - updating schema column descriptions
* DENG-2492 - remove extra space in yaml
* DENG-2492 - Remove unnecessary auto-generated comment
* Adding column descriptions for language and locale
* Adding additional clustering fields
2024-01-29 11:20:45 -06:00
b1d82b1964
Remove deprecation metadata from telemmetry_derived.events_v1 ( #4907 )
2024-01-29 08:50:11 -08:00
3e12c5a3cb
Minor updates to www_site_downloads_v2 DENG-2492 ( #4902 )
...
* Minor updates to www_site_downloads_v2 DENG-2492
* Update schema.yaml
* Update query.sql
* Update query.sql
* Update sql/moz-fx-data-marketing-prod/ga_derived/www_site_downloads_v2/query.sql
Co-authored-by: Sean Rose <1994030+sean-rose@users.noreply.github.com>
* Update query.sql
* Update query.sql
* Fix formatting
---------
Co-authored-by: Sean Rose <1994030+sean-rose@users.noreply.github.com>
2024-01-29 09:02:22 -06:00
cf2a320b06
Delete active_users_aggregates_v1/checks.sql as no longer needed due to the dataset being deprecated ( #4910 )
2024-01-29 15:37:06 +01:00
f995f65bec
Fix table access where dataset ACLs have changed due to deprecation ( #4905 )
2024-01-29 01:33:33 +00:00
073b1f050d
Fix dataset deprecation metadata ( #4874 )
...
* Update dataset workgroup_access when deprecated: true
* Update deprecation metadata tests
* Add metadata.yaml files in telemetry_derived for tables that are managed through other tooling
* Deprecate telemetry_derived datasets
2024-01-26 22:03:54 +00:00
160bfe2ba0
DENG-2492 - Create new GA4 derived table: www_site_downloads_v2 ( #4898 )
...
* DENG-2492 - www_site_downloads_v2 initial commit
* DENG-2492 - add column descriptions
* DENG-2492 - Update metadata.yaml
* DENG-2492 Remove old comments
2024-01-26 15:32:50 -06:00
1b9750cdfb
Don't deploy ETLs with `destination_table: null` set in their scheduling metadata. ( #4897 )
2024-01-26 12:21:45 -08:00
f49ca466f9
Ignore missing SQL generators. ( #4896 )
2024-01-26 11:55:16 -08:00
401b8e7351
Use state_values_v2 for Android LTV pipeline ( #4887 )
...
* Don't try to write existing view files
* Use state_values_v2 for client ad click predictions
* Normalize countries in client_ltv
* Don't get view if unavailable
* Add test for new version of existing table
* Fully qualify tables in view defn
2024-01-26 13:56:39 -05:00
9bd4d568f5
Add ISO 3166 alpha-3 country codes to `static.country_codes_v1`. ( #4894 )
2024-01-26 10:26:23 -08:00
a22221e1fe
Bug 1876010 - Call out the tiny gotcha our analysis made clear ( #4885 )
2024-01-26 10:59:43 +01:00
51acbcdc94
Create new GA4 www_site_hits_v2 derived table ( #4888 )
...
* www_site_hits_v2 initial commit
* Adding additional column descriptions
* Update sql/moz-fx-data-marketing-prod/ga_derived/www_site_hits_v2/query.sql
Co-authored-by: Anna Scholtz <anna@scholtzan.net>
* Update sql/moz-fx-data-marketing-prod/ga_derived/www_site_hits_v2/query.sql
Co-authored-by: Anna Scholtz <anna@scholtzan.net>
* Update sql/moz-fx-data-marketing-prod/ga_derived/www_site_hits_v2/query.sql
Co-authored-by: Anna Scholtz <anna@scholtzan.net>
* Make group by explicit for session_exits CTE
* Update sql/moz-fx-data-marketing-prod/ga_derived/www_site_hits_v2/query.sql
Co-authored-by: Lucia <30448600+lucia-vargas-a@users.noreply.github.com>
* Update sql/moz-fx-data-marketing-prod/ga_derived/www_site_hits_v2/query.sql
Co-authored-by: Lucia <30448600+lucia-vargas-a@users.noreply.github.com>
* Update sql/moz-fx-data-marketing-prod/ga_derived/www_site_hits_v2/metadata.yaml
Co-authored-by: Lucia <30448600+lucia-vargas-a@users.noreply.github.com>
* Update sql/moz-fx-data-marketing-prod/ga_derived/www_site_hits_v2/metadata.yaml
Co-authored-by: Lucia <30448600+lucia-vargas-a@users.noreply.github.com>
* Update sql/moz-fx-data-marketing-prod/ga_derived/www_site_hits_v2/query.sql
Co-authored-by: Lucia <30448600+lucia-vargas-a@users.noreply.github.com>
* Update sql/moz-fx-data-marketing-prod/ga_derived/www_site_hits_v2/query.sql
Co-authored-by: Lucia <30448600+lucia-vargas-a@users.noreply.github.com>
* fix column name
* fix column name
* Added additional column descriptions
* only looking at visits where GA provided a session ID
* Reformatted query file
* Making aliases more meaningful
---------
Co-authored-by: Anna Scholtz <anna@scholtzan.net>
Co-authored-by: Lucia <30448600+lucia-vargas-a@users.noreply.github.com>
2024-01-25 16:55:25 -06:00
eb8c0fb9e2
Run schema updates for new tables using is_init() ( #4890 )
2024-01-25 14:47:27 -08:00
fddb9bbe88
Revert "change 'creation_date' to 'table_creation_date' to be more clear" ( #4889 )
2024-01-25 14:23:19 -08:00
fca4152ed9
Removed validate backfill ci check ( #4883 )
2024-01-25 14:05:53 -08:00
dd8d736c99
change 'creation_date' to 'table_creation_date' to be more clear ( #4797 )
...
* change 'creation_date' to 'table_creation_date' to be more clear
* take out 'creation_date' from bigquery_usage_v2, unnecessary since it is the same as submission_date
2024-01-25 13:56:07 -05:00
395b2689d4
change android onboarding funnel ( #4880 )
...
Co-authored-by: Anna Scholtz <anna@scholtzan.net>
2024-01-24 14:49:00 -08:00
9201c870ce
Make `bqetl query initialize` work for ETLs using `is_init()` in non-default projects. ( #4882 )
2024-01-24 13:24:02 -08:00
a70b2aa689
Support symlinks ( #4881 )
...
* Avoid using `Path.glob()` or `Path.rglob()` for recursive file searches.
Because they don't currently support following symlinks (they will in Python 3.13).
* Specify `followlinks=True` as necessary when calling `os.walk()`.
2024-01-24 13:02:43 -08:00
dfa1208d81
Add android_state_v2 to view ( #4876 )
...
* Add android_state_v2 to view
* Add new state function to client_ltv view
2024-01-24 14:41:59 -05:00
562544690f
Remove generated content from main ( #4867 )
...
* Remove generated content from main
* Fix file permissions and comments
2024-01-24 09:52:04 -08:00
e067426d97
Enable the events stream table for more products ( #4879 )
...
* Enable the events stream table for more products
This enables the events stream table for the Glean Debug Ping Viewer and the Glean Dictionary, in the spirit of dogfooding the table internally a bit more.
* Update bqetl_project.yaml
---------
Co-authored-by: Anna Scholtz <anna@scholtzan.net>
2024-01-24 17:51:19 +01:00
09625d2026
Look for even more FxA deletion events (DENG-2422). ( #4877 )
2024-01-24 07:48:11 -08:00
39de0aea8e
Add table for Android Client LTV ( #4868 )
...
* Add client_ltv table
* Move ltv_state_values to separate materialized table
* Reformat
* Use correct tables for prod
* Use correct dag name
* Use backticks for references
* Run formatter
* Use correct table in view defn
2024-01-24 10:22:17 -05:00
68aa7e54ab
Add android_states_v2 function ( #4875 )
2024-01-24 09:42:57 -05:00
440ba12d77
feat(activity_stream): create new activity stream live view ( #4850 )
...
* feat(activity_stream): in order to update a prefect to using new service account to access live data we need to create a proper live view with access granted to a proper workgroup that the service account will be assigned to
cannot merge until new workgroup is properly setup
* feat: adding new workgroup to dataset
* feat(prefect): adding new workgroup as needed for access to newtab data
* Update sql/moz-fx-data-shared-prod/activity_stream/impression_stats_live/metadata.yaml
Co-authored-by: whd <whd@users.noreply.github.com>
* Update sql/moz-fx-data-shared-prod/firefox_desktop/dataset_metadata.yaml
Co-authored-by: whd <whd@users.noreply.github.com>
* Update sql/moz-fx-data-shared-prod/firefox_desktop/newtab_live/metadata.yaml
Co-authored-by: whd <whd@users.noreply.github.com>
* feat(prefect): updating newtab dataset permissions for prefect
* Use same access regime for pocket/prefect as mozsocial/prefect
---------
Co-authored-by: whd <whd@users.noreply.github.com>
Co-authored-by: Wesley Dawson <whd@mozilla.com>
2024-01-23 23:26:43 +00:00
10d7002b95
Look for additional FxA deletion events (DENG-2422). ( #4870 )
2024-01-23 08:14:29 -08:00
b0387fb7de
Remove referenced_tables usages ( #4834 )
...
* Remove referenced_tables usages
* Resolve check dependencies when generating DAGs
* Add tests for checks automatically resolving dependencies
* Incorporate feedback for removing referenced_tables
* Use depends_on for empty_checks
* More depends_on and fixes
2024-01-22 12:41:05 -08:00
kik-kik
Katie Windau
Eduardo Filho
Anna Scholtz
Jan-Erik Rediger
Curtis Morales
Sean Rose
Winnie Chan
Alexander
dependabot[bot]
whd
Frank Bertsch
Marlene Hirose
Leif Oines
Alessio Placitelli
Braunk