09d1ee7ca5
Add shell completion script for bqetl ( #6372 )
2024-10-21 15:18:16 +00:00
eaf434a834
fix(GLAM): Fix boolean parameter passing to generate_glean_sql ( #6155 )
2024-09-04 11:41:34 -04:00
8b0a3134ee
chore(GLAM): Turn FoG histogram_bucket_counts into a TaskGroup ( #6150 )
2024-09-03 13:59:23 -04:00
b994884098
GLAM purge percentile calculations and prep downstream ( #5966 )
...
* Remove percentiles
* Remove tests that test percentiles
* Refresh scripts insert null to new percentiles
* Remove percentile columns from queries and schemas
* Delete more percentile tables
* Formatting
* histogram_cast_struct's keys are strings
* Re-add test after fixing failure cause
2024-07-25 10:44:43 -04:00
1758ea2df7
chore(glam): remove extract to gcs ( #5780 )
2024-07-08 18:39:52 -04:00
e71fa15800
fix(glam): add a more robust way to fetch glean tables ( #5787 )
...
* fix(glam): add a more robust way to fetch glean tables
* Add jq to image
* Remove useless space
2024-06-13 20:38:41 -04:00
ef947b6b2d
GLAM fix replace_dataset to account for already fully qualified datasets in query ( #5517 )
2024-05-07 18:52:30 -04:00
460ee0f16c
bug 1890351: Move GLAM ETL legacy to glam-prod project
2024-05-06 12:14:08 -04:00
091103703b
GLAM: Definitely fixing grep regex to filter glean tables ( #5239 )
...
* GLAM: Definitely fixing grep regex to filter glean tables
* Remove -E param
2024-03-19 12:12:30 -07:00
2b6d37e288
GLAM try egrep for table fitering + more specific pattern ( #5235 )
...
* GLAM try egrep for table fitering
* More precise label pattern
2024-03-19 13:36:50 -04:00
b6f720d41a
Filter tables using the `glean-min` schema ( #5233 )
...
* Filter tables using the `glean-min` schema
* Update script/glam/generate_glean_sql
Co-authored-by: Eduardo Filho <edugomfilho@gmail.com>
* Update script/glam/generate_glean_sql
update comment
Co-authored-by: Eduardo Filho <edugomfilho@gmail.com>
* Don't freeze the supported schema version
---------
Co-authored-by: Eduardo Filho <edugomfilho@gmail.com>
2024-03-19 10:14:21 -05:00
84ee88e2b9
Dependabot/pip/black 24.1.1 fix ( #5027 )
...
* Bump black from 23.10.1 to 24.1.1
Bumps [black](https://github.com/psf/black ) from 23.10.1 to 24.1.1.
- [Release notes](https://github.com/psf/black/releases )
- [Changelog](https://github.com/psf/black/blob/main/CHANGES.md )
- [Commits](https://github.com/psf/black/compare/23.10.1...24.1.1 )
---
updated-dependencies:
- dependency-name: black
dependency-type: direct:production
update-type: version-update:semver-major
...
Signed-off-by: dependabot[bot] <support@github.com>
* Reformat files with black to fix dependabot update.
* Reformat with black 24.1.1. Update test dag with required space.
* Update test dags.
---------
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-02-19 15:27:34 +01:00
8ec7516157
Issue 4135: Added publish metadata cli command ( #5011 )
...
* Added publish metadata cli command
* Removed publish metadata script
2024-02-12 11:12:14 -08:00
133adfd472
Don't overwrite existing `gcp.json` files unless that's actually necessary ( #4997 )
2024-02-08 15:37:28 -08:00
e2f33ed29b
Add debug messages for `gcp.json` file. ( #4979 )
2024-02-07 08:12:25 -08:00
451310a033
Create unique `gcp.json` temp files to avoid file contention from parallel tests. ( #4924 )
2024-01-30 16:03:16 -08:00
1d1addb86c
Indent join conditions ( #4223 )
...
* Indent join conditions.
* Put parentheses around some `BETWEEN ... AND ...` join conditions.
2024-01-11 15:50:26 -08:00
82308b91d5
Bug 1866469 - Exclude use_counters from GLAM ETL ( #4603 )
...
* Bug 1866469 - Exclude use_counters from GLAM ETL
* Attempt to fix tests
---------
Co-authored-by: Eduardo Filho <edugomfilho@gmail.com>
2023-11-24 16:00:24 -05:00
ae68eec512
added a debug message and flag for when authenticating to gcloud ( #3602 )
2023-02-23 18:43:24 +01:00
c281400486
Enforce isort via pytest ( #3384 )
2022-11-30 11:45:05 -08:00
3b03c1db10
Move copy_deduplicate to CLI
2022-09-06 14:22:17 -07:00
81fbdf6c01
Move doc validation and generation commands to CLI
2022-08-25 08:51:41 -07:00
ebc6c0d0ab
Move list_broken_views to CLI
2022-08-24 09:01:29 -07:00
2c1f294445
Remove json_to_table_ddl
2022-08-24 09:01:29 -07:00
d95f844002
Move static publish script to CLI
2022-08-23 14:54:23 -07:00
a54e3f7309
Remove old run scripts
2022-08-17 12:58:52 -07:00
8ddf96b3ff
Remove scripts that have been replaced by CLI
2022-08-17 12:58:52 -07:00
9a5a98779a
Remove scripts that have been replaced by CLI
2022-07-19 08:08:49 -07:00
0ca902fe51
Bump google-cloud-bigquery from 2.34.2 to 3.2.0 ( #3020 )
2022-06-15 18:36:18 +00:00
5ea2f1dea2
Remove redundant GOOGLE_APPLICATION_CREDENTIALS in integration ( #2973 )
2022-05-16 19:53:50 +00:00
a72ecf7f59
GLAM ETL: filter out unofficial desktop builds in FOG pipeline ( #2957 )
...
GLAM ETL: filter out unofficial desktop builds in FOG pipeline
In https://github.com/mozilla/glam/issues/1941 it was discovered that some desktop clients send version "1024" in FOG telemetry. This is breaking GLAM aggregations because they focus on latest released versions. In order to address that, from now on we'll:
1. filter pings by build_id against official Mozilla builds published in Buildhub. This is equivalent to the filtering we already do in the legacy telemetry GLAM ETL: 9bca48821a/sql/moz-fx-data-shared-prod/telemetry_derived/clients_daily_keyed_scalar_aggregates_v1/query.sql (L3-L8)https://bugzilla.mozilla.org/show_bug.cgi?id=1768187 for details)
2022-05-11 00:33:35 +02:00
e5dd8f8aba
Bug 1757216 Add ISP field to baseline_clients_daily ( #2928 )
...
* Bug 1757216 Add ISP field to baseline_clients_daily
Replaces #2919
This will enable use to filter out clients sent by BrowserStack in downstream tables and queries.
* Add refs to PR and bug
* Remove test table in script
Co-authored-by: Anna Scholtz <anna@scholtzan.net>
Co-authored-by: Anna Scholtz <anna@scholtzan.net>
2022-05-02 12:04:28 -04:00
a2693b8f98
Glam-ETL: print out script statements
2022-04-28 16:55:45 +02:00
c590cb3b70
Bump click from 8.1.0 to 8.1.1 ( #2849 )
2022-03-31 10:51:51 -07:00
7998cafaab
Bump click from 8.0.4 to 8.1.0 ( #2838 )
2022-03-29 20:46:31 +02:00
a436a574ff
Add agg_type to the sample counts for desktop and glean, remove the extract sample counts query ( #2772 )
...
* added agg_type for sample counts table
* removed the extract sample counts for both dekstop and glean products
* corrected sql formatting
corrected sql formatting
2022-03-02 15:51:20 -05:00
4406437be2
Use --no-deps when installing compiled requirements files ( #2752 )
2022-02-24 21:36:47 +00:00
9759c46510
Make sure generate_sql script is compatible with cloud pipeline
2022-02-16 10:00:16 -08:00
b5f9ea4752
Update generate_sql script
2022-02-16 08:52:51 -08:00
588364b9aa
Revert "adding 2m sleep for the export task to complete ( #2609 )" ( #2660 )
...
This reverts commit d993e735fe
2022-01-14 12:54:10 -05:00
45655229d3
Added sql_generators script to create schema.yaml files for derived views ( #2657 )
2022-01-13 15:46:06 -05:00
678c5b2a6c
Update glean_usage generate parameter in scripts
2022-01-07 15:07:12 -08:00
cf966d2280
Move stable view generation into separate module
2022-01-05 12:26:52 -08:00
2b3eebe812
Update scripts and test for sql_generators/
2022-01-03 14:15:02 -08:00
d993e735fe
adding 2m sleep for the export task to complete ( #2609 )
2022-01-03 12:13:02 -05:00
80af7b96df
firefox_desktop_to_glam ( #2485 )
...
* firefox_desktop_to_glam
added sql files part 1
* update with the lastest mozfun function
2021-11-22 13:45:19 -05:00
4a4ba71d7f
Remove references to moz-fx-data-derived-datasets ( #2490 )
2021-11-11 18:28:32 +00:00
03c38dc203
Remove -n from entrypoint scripts
2021-11-08 10:29:14 -08:00
6200c5ed36
Run SQL tests separately
2021-11-08 10:29:14 -08:00
98fb5fb2cf
Speed up build
2021-11-08 10:29:14 -08:00
Ben Wu
Eduardo Filho
wil stuckey
Lucia
Winnie Chan
Sean Rose
akkomar
kik-kik
Daniel Thorn
Anna Scholtz
dependabot[bot]
Jeff Klukas
Arkadiusz Komarzewski
Alekhya
Alexander Nicholson
whd