mozilla
/
bigquery-etl
зеркало из https://github.com/mozilla/bigquery-etl.git
1
0
Форкнуть 0
Код Задачи Пакеты Проекты Релизы Вики Активность
5 861 коммит 321 ветка 1 297 Теги 223 MiB
Граф коммитов

454 Коммитов

Автор SHA1 Сообщение Дата
Eduardo Filho b994884098
GLAM purge percentile calculations and prep downstream (#5966) 
* Remove percentiles

* Remove tests that test percentiles

* Refresh scripts insert null to new percentiles

* Remove percentile columns from queries and schemas

* Delete more percentile tables

* Formatting

* histogram_cast_struct's keys are strings

* Re-add test after fixing failure cause
 2024-07-25 10:44:43 -04:00
Eduardo Filho 1758ea2df7
chore(glam): remove extract to gcs (#5780) 2024-07-08 18:39:52 -04:00
Eduardo Filho e71fa15800
fix(glam): add a more robust way to fetch glean tables (#5787) 
* fix(glam): add a more robust way to fetch glean tables

* Add jq to image

* Remove useless space
 2024-06-13 20:38:41 -04:00
Eduardo Filho ef947b6b2d
GLAM fix replace_dataset to account for already fully qualified datasets in query (#5517) 2024-05-07 18:52:30 -04:00
Eduardo Filho 460ee0f16c
bug 1890351: Move GLAM ETL legacy to glam-prod project 2024-05-06 12:14:08 -04:00
Eduardo Filho 091103703b
GLAM: Definitely fixing grep regex to filter glean tables (#5239) 
* GLAM: Definitely fixing grep regex to filter glean tables

* Remove -E param
 2024-03-19 12:12:30 -07:00
Eduardo Filho 2b6d37e288
GLAM try egrep for table fitering + more specific pattern (#5235) 
* GLAM try egrep for table fitering

* More precise label pattern
 2024-03-19 13:36:50 -04:00
wil stuckey b6f720d41a
Filter tables using the `glean-min` schema (#5233) 
* Filter tables using the `glean-min` schema

* Update script/glam/generate_glean_sql

Co-authored-by: Eduardo Filho <edugomfilho@gmail.com>

* Update script/glam/generate_glean_sql

update comment

Co-authored-by: Eduardo Filho <edugomfilho@gmail.com>

* Don't freeze the supported schema version

---------

Co-authored-by: Eduardo Filho <edugomfilho@gmail.com>
 2024-03-19 10:14:21 -05:00
Lucia 84ee88e2b9
Dependabot/pip/black 24.1.1 fix (#5027) 
* Bump black from 23.10.1 to 24.1.1

Bumps [black](https://github.com/psf/black) from 23.10.1 to 24.1.1.
- [Release notes](https://github.com/psf/black/releases)
- [Changelog](https://github.com/psf/black/blob/main/CHANGES.md)
- [Commits](https://github.com/psf/black/compare/23.10.1...24.1.1)

---
updated-dependencies:
- dependency-name: black
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>

* Reformat files with black to fix dependabot update.

* Reformat with black 24.1.1. Update test dag with required space.

* Update test dags.

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2024-02-19 15:27:34 +01:00
Winnie Chan 8ec7516157
Issue 4135: Added publish metadata cli command (#5011) 
* Added publish metadata cli command

* Removed publish metadata script
 2024-02-12 11:12:14 -08:00
Sean Rose 133adfd472
Don't overwrite existing `gcp.json` files unless that's actually necessary (#4997) 2024-02-08 15:37:28 -08:00
Sean Rose e2f33ed29b
Add debug messages for `gcp.json` file. (#4979) 2024-02-07 08:12:25 -08:00
Sean Rose 451310a033
Create unique `gcp.json` temp files to avoid file contention from parallel tests. (#4924) 2024-01-30 16:03:16 -08:00
Sean Rose 1d1addb86c
Indent join conditions (#4223) 
* Indent join conditions.

* Put parentheses around some `BETWEEN ... AND ...` join conditions.
 2024-01-11 15:50:26 -08:00
akkomar 82308b91d5
Bug 1866469 - Exclude use_counters from GLAM ETL (#4603) 
* Bug 1866469 - Exclude use_counters from GLAM ETL

* Attempt to fix tests

---------

Co-authored-by: Eduardo Filho <edugomfilho@gmail.com>
 2023-11-24 16:00:24 -05:00
kik-kik ae68eec512
added a debug message and flag for when authenticating to gcloud (#3602) 2023-02-23 18:43:24 +01:00
Daniel Thorn c281400486
Enforce isort via pytest (#3384) 2022-11-30 11:45:05 -08:00
Anna Scholtz 3b03c1db10 Move copy_deduplicate to CLI 2022-09-06 14:22:17 -07:00
Anna Scholtz 81fbdf6c01 Move doc validation and generation commands to CLI 2022-08-25 08:51:41 -07:00
Anna Scholtz ebc6c0d0ab Move list_broken_views to CLI 2022-08-24 09:01:29 -07:00
Anna Scholtz 2c1f294445 Remove json_to_table_ddl 2022-08-24 09:01:29 -07:00
Anna Scholtz d95f844002 Move static publish script to CLI 2022-08-23 14:54:23 -07:00
Anna Scholtz a54e3f7309 Remove old run scripts 2022-08-17 12:58:52 -07:00
Anna Scholtz 8ddf96b3ff Remove scripts that have been replaced by CLI 2022-08-17 12:58:52 -07:00
Anna Scholtz 9a5a98779a Remove scripts that have been replaced by CLI 2022-07-19 08:08:49 -07:00
dependabot[bot] 0ca902fe51
Bump google-cloud-bigquery from 2.34.2 to 3.2.0 (#3020) 2022-06-15 18:36:18 +00:00
Daniel Thorn 5ea2f1dea2
Remove redundant GOOGLE_APPLICATION_CREDENTIALS in integration (#2973) 2022-05-16 19:53:50 +00:00
akkomar a72ecf7f59
GLAM ETL: filter out unofficial desktop builds in FOG pipeline (#2957) 
GLAM ETL: filter out unofficial desktop builds in FOG pipeline

In https://github.com/mozilla/glam/issues/1941 it was discovered that some desktop clients send version "1024" in FOG telemetry. This is breaking GLAM aggregations because they focus on latest released versions. In order to address that, from now on we'll:
1. filter pings by build_id against official Mozilla builds published in Buildhub. This is equivalent to the filtering we already do in the legacy telemetry GLAM ETL: 9bca48821a/sql/moz-fx-data-shared-prod/telemetry_derived/clients_daily_keyed_scalar_aggregates_v1/query.sql (L3-L8)
2. Explicitly filter out pings with version "1024.0.0". At the moment we do not know why this version number is sent by clients (see https://bugzilla.mozilla.org/show_bug.cgi?id=1768187 for details)
 2022-05-11 00:33:35 +02:00
Jeff Klukas e5dd8f8aba
Bug 1757216 Add ISP field to baseline_clients_daily (#2928) 
* Bug 1757216 Add ISP field to baseline_clients_daily

Replaces #2919

This will enable use to filter out clients sent by BrowserStack in downstream tables and queries.

* Add refs to PR and bug

* Remove test table in script

Co-authored-by: Anna Scholtz <anna@scholtzan.net>

Co-authored-by: Anna Scholtz <anna@scholtzan.net>
 2022-05-02 12:04:28 -04:00
Arkadiusz Komarzewski a2693b8f98 Glam-ETL: print out script statements 2022-04-28 16:55:45 +02:00
dependabot[bot] c590cb3b70
Bump click from 8.1.0 to 8.1.1 (#2849) 2022-03-31 10:51:51 -07:00
dependabot[bot] 7998cafaab
Bump click from 8.0.4 to 8.1.0 (#2838) 2022-03-29 20:46:31 +02:00
Alekhya a436a574ff
Add agg_type to the sample counts for desktop and glean, remove the extract sample counts query (#2772) 
* added agg_type for sample counts table

* removed the extract sample counts for both dekstop and glean products

* corrected sql formatting

corrected sql formatting
 2022-03-02 15:51:20 -05:00
Daniel Thorn 4406437be2
Use --no-deps when installing compiled requirements files (#2752) 2022-02-24 21:36:47 +00:00
Anna Scholtz 9759c46510 Make sure generate_sql script is compatible with cloud pipeline 2022-02-16 10:00:16 -08:00
Anna Scholtz b5f9ea4752 Update generate_sql script 2022-02-16 08:52:51 -08:00
Alekhya 588364b9aa
Revert "adding 2m sleep for the export task to complete (#2609)" (#2660) 
This reverts commit d993e735fe.
 2022-01-14 12:54:10 -05:00
Alexander Nicholson 45655229d3
Added sql_generators script to create schema.yaml files for derived views (#2657) 2022-01-13 15:46:06 -05:00
Anna Scholtz 678c5b2a6c Update glean_usage generate parameter in scripts 2022-01-07 15:07:12 -08:00
Anna Scholtz cf966d2280 Move stable view generation into separate module 2022-01-05 12:26:52 -08:00
Anna Scholtz 2b3eebe812 Update scripts and test for sql_generators/ 2022-01-03 14:15:02 -08:00
Alekhya d993e735fe
adding 2m sleep for the export task to complete (#2609) 2022-01-03 12:13:02 -05:00
Alekhya 80af7b96df
firefox_desktop_to_glam (#2485) 
* firefox_desktop_to_glam

added sql files part 1

* update with the lastest mozfun function
 2021-11-22 13:45:19 -05:00
whd 4a4ba71d7f
Remove references to moz-fx-data-derived-datasets (#2490) 2021-11-11 18:28:32 +00:00
Anna Scholtz 03c38dc203 Remove -n from entrypoint scripts 2021-11-08 10:29:14 -08:00
Anna Scholtz 6200c5ed36 Run SQL tests separately 2021-11-08 10:29:14 -08:00
Anna Scholtz 98fb5fb2cf Speed up build 2021-11-08 10:29:14 -08:00
Anthony Miyaguchi 94d0003f16
Add minor modifications to glam docs (#2342) 
* Add minor modifications to glam docs

* Update about checking in to repo

Co-authored-by: Alekhya <88394696+alekhyamoz@users.noreply.github.com>
 2021-11-02 12:17:02 -04:00
Alekhya df5eb5e77e
Added sample counts for glam fenix (#2355) 
* added sample counts for glam fenix

* formatted for black format check

* Revert "formatted for black format check"

This reverts commit cf71fed487.

* formated for black format check

* added the sample coubt scripts
 2021-09-21 16:18:41 -04:00
Will Lachance 411af312e4
Replace `format_sql` with "bqetl format" (#2348) 
* Make bqetl work equivalently to format_sql (which had some extra
options to format standard in)
* Remove `format_sql` and update everything that uses it to use
  `bqetl format` instead.
 2021-09-20 17:18:25 +00:00