f500ba9b7a
DENG-1705 - Add missing client attribution columns to clients daily/first-seen ( #4505 )
...
* DENG-1705 Add missing client attribution columns to clients daily/firstseen
* Update clients_last_seen_joined
2023-11-08 15:13:17 -05:00
2ca7b4be76
Fix comments in event monitoring queries ( #4535 )
2023-11-08 19:21:52 +01:00
1be564e23c
DENG-1935 Change data ordering from pings in clients-first-seen-v2 ( #4533 )
...
* DENG-1935 Change data ordering from pings in clients-first-seen-v2
* Added main ping for client-3, maintain chosen ping
2023-11-08 10:49:17 -05:00
bfbae901fc
Add ios baseline_clients_yearly ( #4506 )
2023-11-08 10:24:52 -05:00
894d42dde1
DS-3054. Support running an initialization query in parallel ( #4322 )
...
* DS-3054. Create functions to support running an initialization query for all sample_ids in parallel.
* DS-3054. Update _run_query function.
* DS-3054. Use _run_query and mapped values for initialization in parallel.
* DS-3054. Unify initialization to run in parallel and get sample_id range from metadata.
* DS-3054. Minimize formatting of query template and remove need to modify existing initialization queries. Validate if a query should use parallelized or regular update.
* DS-3054. Adding link to caveats.
* DS-3054. Update sample_id range for initialization.
* DS-3054. Use current implementation of run_query.
* DS-3054. Update using a parameter instead of initialization in metadata.
* DS-3054. DAG update with new parameter.
* Pass parameters before calling _run_query().
* Use --append_tablein favour of INSERT INTO.
* DS-3054 Separate parallel and non parallel init, plus some improvements.
---------
Co-authored-by: Lucia Vargas <lvargas@mozilla.com>
2023-11-07 20:03:48 +01:00
032dc28025
Add submission_date param to adclicks history ( #4531 )
2023-11-07 11:58:59 -05:00
65f7f55848
feat(DENG-1774): adding fenix derived firefox android clients v2 ( #4424 )
...
* added fenix_derirved.firefox_android_clients_v2
* added ETL checks for fenix_derirved.firefox_android_clients_v2
* made changes as suggested by bani in PR#4424
* converting unique check for android clients v2 until duplication is resolved
* added install_source field to firefox_android_clients_v2 and formatting applied on checks
* added locale field and modified the query to suppot is_init()
* removed generated dag due to new generation process
2023-11-07 10:39:37 +01:00
0ae76d0971
Add experiment information to event monitoring ( #4519 )
2023-11-06 16:13:02 -08:00
3915be19ee
DS-3281 - Add client adclicks history table ( #4528 )
...
* Add client adclicks history table
* Add alias to ad_click_history col
Co-authored-by: Anna Scholtz <anna@scholtzan.net>
* Remove partition parameter on table write
---------
Co-authored-by: Anna Scholtz <anna@scholtzan.net>
2023-11-06 16:25:17 -05:00
e0eb33509b
Add map.set_key to mozfun ( #4527 )
...
* Add map.set_key to mozfun
* Disallow NULL keys in maps
2023-11-06 14:44:03 -05:00
e7e7eaae06
Set depend_on_past=False for warn checks ( #4526 )
2023-11-06 10:39:58 -08:00
b3abfc42ce
Update scheduler of aggregates to run after upstreams. ( #4503 )
...
* Update scheduler of aggregates to run after upstreams.
* Update dags for new scheduler of analytics_aggregates
* Update dag bqetl_search
* Remove DAG.
---------
Co-authored-by: Lucia Vargas <lvargas@mozilla.com>
2023-11-06 17:55:12 +01:00
a30788cdd5
updated schema for telemetry_derived/clients_last_seen_joined_v1 to align it with the query results ( #4523 )
2023-11-06 12:38:15 +01:00
0962ba65fe
prefixing schema error message inside dryrun to "ERROR" to make it easier to find when searching logs for cause of exit code 1 ( #4522 )
2023-11-06 12:12:50 +01:00
0c021e700a
Add query to load application information from probe info service ( #4508 )
2023-11-03 11:11:54 -07:00
a271c024b2
Dont generate dags in bqetl query schedule command ( #4517 )
2023-11-03 08:59:27 -07:00
69259c63a4
Bump mkdocs-material from 9.3.1 to 9.4.7 ( #4518 )
...
Bumps [mkdocs-material](https://github.com/squidfunk/mkdocs-material ) from 9.3.1 to 9.4.7.
- [Release notes](https://github.com/squidfunk/mkdocs-material/releases )
- [Changelog](https://github.com/squidfunk/mkdocs-material/blob/master/CHANGELOG )
- [Commits](https://github.com/squidfunk/mkdocs-material/compare/9.3.1...9.4.7 )
---
updated-dependencies:
- dependency-name: mkdocs-material
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-11-03 15:08:56 +01:00
3883120d24
Bump pip from 23.0 to 23.3 ( #4516 )
...
Bumps [pip](https://github.com/pypa/pip ) from 23.0 to 23.3.
- [Changelog](https://github.com/pypa/pip/blob/main/NEWS.rst )
- [Commits](https://github.com/pypa/pip/compare/23.0...23.3 )
---
updated-dependencies:
- dependency-name: pip
dependency-type: indirect
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-11-03 14:41:25 +01:00
9780c0e9fb
Remove distinct_docids query ( #4449 )
2023-11-02 13:37:38 -07:00
2c5ca38e81
Fix funnel generation logic ( #4514 )
2023-11-02 13:19:40 -07:00
fb810d13c0
Bump pre-commit from 3.4.0 to 3.5.0 ( #4510 )
...
Bumps [pre-commit](https://github.com/pre-commit/pre-commit ) from 3.4.0 to 3.5.0.
- [Release notes](https://github.com/pre-commit/pre-commit/releases )
- [Changelog](https://github.com/pre-commit/pre-commit/blob/main/CHANGELOG.md )
- [Commits](https://github.com/pre-commit/pre-commit/compare/v3.4.0...v3.5.0 )
---
updated-dependencies:
- dependency-name: pre-commit
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-11-02 18:40:19 +01:00
3619240028
Bump pyarrow from 13.0.0 to 14.0.0 ( #4511 )
...
Bumps [pyarrow](https://github.com/apache/arrow ) from 13.0.0 to 14.0.0.
- [Commits](https://github.com/apache/arrow/compare/go/v13.0.0...go/v14.0.0 )
---
updated-dependencies:
- dependency-name: pyarrow
dependency-type: direct:production
update-type: version-update:semver-major
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-11-02 17:17:03 +01:00
3483a74844
Add output_dir to command dag generate. ( #4512 )
...
* Add output_dir to command dag generate.
* output_dir to command dag generate.
* output_dir to command dag generate.
---------
Co-authored-by: Lucia Vargas <lvargas@mozilla.com>
2023-11-02 16:30:29 +01:00
dffbb3afef
Remove generated DAGs from main ( #4507 )
2023-11-01 15:05:59 -07:00
185f833f2a
Materialized views and aggregated tables for event monitoring ( #4478 )
...
* WIP event monitoring
* Add FxA custom events to view definition (#4483 )
* Add FxA custom events to view definition
* Update sql_generators/event_monitoring/templates/event_monitoring_live.init.sql
* Update sql_generators/event_monitoring/templates/event_monitoring_live.init.sql
* Update sql_generators/event_monitoring/templates/event_monitoring_live.init.sql
* Update sql_generators/event_monitoring/templates/event_monitoring_live.init.sql
---------
Co-authored-by: Anna Scholtz <anna@scholtzan.net>
* Move event monitoring to glean_usage generator
* Add cross-app event monitoring view
* Generate cross app monitoring
* Simplyfy event monitoring aggregation
---------
Co-authored-by: akkomar <akkomar@users.noreply.github.com>
2023-11-01 14:20:20 -07:00
1e42477c38
Bump sqlglot from 18.11.4 to 19.0.1 ( #4500 )
...
Bumps [sqlglot](https://github.com/tobymao/sqlglot ) from 18.11.4 to 19.0.1.
- [Changelog](https://github.com/tobymao/sqlglot/blob/main/CHANGELOG.md )
- [Commits](https://github.com/tobymao/sqlglot/compare/v18.11.4...v19.0.1 )
---
updated-dependencies:
- dependency-name: sqlglot
dependency-type: direct:production
update-type: version-update:semver-major
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-11-01 12:35:11 -04:00
a699decdbe
Bump black from 23.9.1 to 23.10.1
...
Bumps [black](https://github.com/psf/black ) from 23.9.1 to 23.10.1.
- [Release notes](https://github.com/psf/black/releases )
- [Changelog](https://github.com/psf/black/blob/main/CHANGES.md )
- [Commits](https://github.com/psf/black/compare/23.9.1...23.10.1 )
---
updated-dependencies:
- dependency-name: black
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
2023-11-01 17:18:23 +01:00
9786d377e8
Fix test for norm.glean_ping_info
2023-11-01 16:58:00 +01:00
326b4812ad
Remove mmccorquodale from DAG owners ( #4492 )
2023-11-01 10:44:21 -04:00
b592df7b5b
Use mozfun.glean.parse_datetime to parse ping_info fields ( #4464 )
...
In future versions of Glean that timestamp can be more precise, so we
need to ensure we correctly parse it.
Co-authored-by: Anna Scholtz <anna@scholtzan.net>
2023-10-31 09:25:15 -07:00
5bc2baf7a4
Enforce no date partition parameter in DAG ( #4497 )
2023-10-31 10:47:32 -05:00
ed6c72b2a3
Bump pytest from 7.4.2 to 7.4.3 ( #4496 )
...
Bumps [pytest](https://github.com/pytest-dev/pytest ) from 7.4.2 to 7.4.3.
- [Release notes](https://github.com/pytest-dev/pytest/releases )
- [Changelog](https://github.com/pytest-dev/pytest/blob/main/CHANGELOG.rst )
- [Commits](https://github.com/pytest-dev/pytest/compare/7.4.2...7.4.3 )
---
updated-dependencies:
- dependency-name: pytest
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-10-31 13:11:40 +01:00
e689f27a75
Bump jsonschema from 4.19.0 to 4.19.2 ( #4495 )
...
Bumps [jsonschema](https://github.com/python-jsonschema/jsonschema ) from 4.19.0 to 4.19.2.
- [Release notes](https://github.com/python-jsonschema/jsonschema/releases )
- [Changelog](https://github.com/python-jsonschema/jsonschema/blob/main/CHANGELOG.rst )
- [Commits](https://github.com/python-jsonschema/jsonschema/compare/v4.19.0...v4.19.2 )
---
updated-dependencies:
- dependency-name: jsonschema
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: akkomar <akkomar@users.noreply.github.com>
2023-10-31 12:29:43 +01:00
2754d7d7c0
DENG-1879 Setup import of emails table from FxA prod CloudSQL ( #4494 )
2023-10-31 10:31:27 +01:00
511894d181
DENG-1879 Setup import of emails table from FxA stage CloudSQL ( #4493 )
2023-10-31 09:12:56 +01:00
c6ffd9e1fd
Glean server knobs monitoring table ( #4491 )
...
* Glean server knobs monitoring table
* fix code gen and skip dry-run
* Remove view creation in query
2023-10-30 16:23:46 -05:00
a647e1f2b3
Bump mozilla-metric-config-parser from 2023.9.2 to 2023.10.2 ( #4476 )
...
Bumps [mozilla-metric-config-parser](https://github.com/mozilla/metric-config-parser ) from 2023.9.2 to 2023.10.2.
- [Release notes](https://github.com/mozilla/metric-config-parser/releases )
- [Commits](https://github.com/mozilla/metric-config-parser/compare/2023.9.2...2023.10.2 )
---
updated-dependencies:
- dependency-name: mozilla-metric-config-parser
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Anna Scholtz <anna@scholtzan.net>
2023-10-30 08:31:59 -07:00
90df953d46
Bump types-requests from 2.31.0.2 to 2.31.0.10 ( #4475 )
...
Bumps [types-requests](https://github.com/python/typeshed ) from 2.31.0.2 to 2.31.0.10.
- [Commits](https://github.com/python/typeshed/commits )
---
updated-dependencies:
- dependency-name: types-requests
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-10-30 08:16:02 -07:00
a42cf83558
Include all sql_gen files in package ( #4490 )
...
When the bigquery-etl package is installed from pypi (or locally
via `pip install .`), the only non-py files included in the package
are those in the `package_data` section of setup.py.
Previously, with just those files, sql generation would fail due
to missing files. Because this directory is small, we should
include all files so no one accidentally runs into this problem
again.
Co-authored-by: Daniel Thorn <dthorn@mozilla.com>
2023-10-27 16:12:16 -04:00
28e7dbfa12
DENG-1781- Remove urlbar_events_temp_v2 view and repoint urlbar_events view to v2 ( #4486 )
...
* Remove urlbar_events_temp_v2 view and repoint urlbar_events view to v2
2023-10-27 14:19:19 -04:00
799aa62af7
fix for #4481 ( #4489 )
2023-10-27 13:52:01 +02:00
61a6b0a77c
Bug 1860814 - fix amo_prod__desktop_addons_by_client ( #4481 )
...
* quick fix
* fix spread out groupby
* move out sourcetable query
---------
Co-authored-by: Frank Bertsch <frank.bertsch@gmail.com>
2023-10-27 12:06:45 +02:00
71e781da97
Fix test for mozfun.norm.result_type_to_product_name ( #4487 )
2023-10-26 10:50:37 -07:00
a66767cac9
DENG-476 - Update internet outages to reference main_v5 ( #4432 )
2023-10-26 10:34:37 -07:00
a9eb4a8242
DENG-476 - Update experiment aggregates ETL to reference main_v5 ( #4435 )
2023-10-26 10:12:19 -07:00
57670281e1
DENG-476 - Update sampled main ping tables to reference main_v5 ( #4433 )
2023-10-26 09:44:57 -07:00
5b2894e876
DENG-476 - Update monitoring ETLs to reference main_v5 ( #4431 )
2023-10-26 09:17:22 -07:00
fac37452ab
Update experiment export query to include feature ids and branch feature config values ( #4477 )
...
* Update experiment export query to include feature ids and branch feature
config value.
* Add view skip for broken view
* add skip to dry run as well
2023-10-26 05:11:06 -07:00
a03d875019
DENG-1705 - Add startup_profile_selection_reason_first to clients_daily_v6 ( #4473 )
2023-10-25 13:40:14 -04:00
73a5535b67
Urlbar events: nested (long) instead of wide ( #4373 )
...
* feat: urlbar events final release
* feat: new result types
* feat: add interaction and group
* fix: date
* fix: use BQ builtin for UUIDs
* Add the view_v2'
* Add new table to the DAG
* fix CI error
fix ci error
* remove teon brooks
* Incorporate feedback by Curtis
Incorporate feedback from Curtis
---------
Co-authored-by: Alekhya Kommasani <akommasani@mozilla.com>
Co-authored-by: Alekhya <88394696+alekhyamoz@users.noreply.github.com>
2023-10-25 12:55:00 -04:00
Alexander
akkomar
Frank Bertsch
Lucia
kik-kik
Anna Scholtz
dependabot[bot]
Jan-Erik Rediger
Curtis Morales
wil stuckey
Alekhya
Leli
Daniel Thorn
Rebecca BurWei