c281400486
Enforce isort via pytest ( #3384 )
2022-11-30 11:45:05 -08:00
3b03c1db10
Move copy_deduplicate to CLI
2022-09-06 14:22:17 -07:00
81fbdf6c01
Move doc validation and generation commands to CLI
2022-08-25 08:51:41 -07:00
ebc6c0d0ab
Move list_broken_views to CLI
2022-08-24 09:01:29 -07:00
2c1f294445
Remove json_to_table_ddl
2022-08-24 09:01:29 -07:00
d95f844002
Move static publish script to CLI
2022-08-23 14:54:23 -07:00
a54e3f7309
Remove old run scripts
2022-08-17 12:58:52 -07:00
8ddf96b3ff
Remove scripts that have been replaced by CLI
2022-08-17 12:58:52 -07:00
9a5a98779a
Remove scripts that have been replaced by CLI
2022-07-19 08:08:49 -07:00
0ca902fe51
Bump google-cloud-bigquery from 2.34.2 to 3.2.0 ( #3020 )
2022-06-15 18:36:18 +00:00
5ea2f1dea2
Remove redundant GOOGLE_APPLICATION_CREDENTIALS in integration ( #2973 )
2022-05-16 19:53:50 +00:00
a72ecf7f59
GLAM ETL: filter out unofficial desktop builds in FOG pipeline ( #2957 )
...
GLAM ETL: filter out unofficial desktop builds in FOG pipeline
In https://github.com/mozilla/glam/issues/1941 it was discovered that some desktop clients send version "1024" in FOG telemetry. This is breaking GLAM aggregations because they focus on latest released versions. In order to address that, from now on we'll:
1. filter pings by build_id against official Mozilla builds published in Buildhub. This is equivalent to the filtering we already do in the legacy telemetry GLAM ETL: 9bca48821a/sql/moz-fx-data-shared-prod/telemetry_derived/clients_daily_keyed_scalar_aggregates_v1/query.sql (L3-L8)https://bugzilla.mozilla.org/show_bug.cgi?id=1768187 for details)
2022-05-11 00:33:35 +02:00
e5dd8f8aba
Bug 1757216 Add ISP field to baseline_clients_daily ( #2928 )
...
* Bug 1757216 Add ISP field to baseline_clients_daily
Replaces #2919
This will enable use to filter out clients sent by BrowserStack in downstream tables and queries.
* Add refs to PR and bug
* Remove test table in script
Co-authored-by: Anna Scholtz <anna@scholtzan.net>
Co-authored-by: Anna Scholtz <anna@scholtzan.net>
2022-05-02 12:04:28 -04:00
a2693b8f98
Glam-ETL: print out script statements
2022-04-28 16:55:45 +02:00
c590cb3b70
Bump click from 8.1.0 to 8.1.1 ( #2849 )
2022-03-31 10:51:51 -07:00
7998cafaab
Bump click from 8.0.4 to 8.1.0 ( #2838 )
2022-03-29 20:46:31 +02:00
a436a574ff
Add agg_type to the sample counts for desktop and glean, remove the extract sample counts query ( #2772 )
...
* added agg_type for sample counts table
* removed the extract sample counts for both dekstop and glean products
* corrected sql formatting
corrected sql formatting
2022-03-02 15:51:20 -05:00
4406437be2
Use --no-deps when installing compiled requirements files ( #2752 )
2022-02-24 21:36:47 +00:00
9759c46510
Make sure generate_sql script is compatible with cloud pipeline
2022-02-16 10:00:16 -08:00
b5f9ea4752
Update generate_sql script
2022-02-16 08:52:51 -08:00
588364b9aa
Revert "adding 2m sleep for the export task to complete ( #2609 )" ( #2660 )
...
This reverts commit d993e735fe
2022-01-14 12:54:10 -05:00
45655229d3
Added sql_generators script to create schema.yaml files for derived views ( #2657 )
2022-01-13 15:46:06 -05:00
678c5b2a6c
Update glean_usage generate parameter in scripts
2022-01-07 15:07:12 -08:00
cf966d2280
Move stable view generation into separate module
2022-01-05 12:26:52 -08:00
2b3eebe812
Update scripts and test for sql_generators/
2022-01-03 14:15:02 -08:00
d993e735fe
adding 2m sleep for the export task to complete ( #2609 )
2022-01-03 12:13:02 -05:00
80af7b96df
firefox_desktop_to_glam ( #2485 )
...
* firefox_desktop_to_glam
added sql files part 1
* update with the lastest mozfun function
2021-11-22 13:45:19 -05:00
4a4ba71d7f
Remove references to moz-fx-data-derived-datasets ( #2490 )
2021-11-11 18:28:32 +00:00
03c38dc203
Remove -n from entrypoint scripts
2021-11-08 10:29:14 -08:00
6200c5ed36
Run SQL tests separately
2021-11-08 10:29:14 -08:00
98fb5fb2cf
Speed up build
2021-11-08 10:29:14 -08:00
94d0003f16
Add minor modifications to glam docs ( #2342 )
...
* Add minor modifications to glam docs
* Update about checking in to repo
Co-authored-by: Alekhya <88394696+alekhyamoz@users.noreply.github.com>
2021-11-02 12:17:02 -04:00
df5eb5e77e
Added sample counts for glam fenix ( #2355 )
...
* added sample counts for glam fenix
* formatted for black format check
* Revert "formatted for black format check"
This reverts commit cf71fed487
2021-09-21 16:18:41 -04:00
411af312e4
Replace `format_sql` with "bqetl format" ( #2348 )
...
* Make bqetl work equivalently to format_sql (which had some extra
options to format standard in)
* Remove `format_sql` and update everything that uses it to use
`bqetl format` instead.
2021-09-20 17:18:25 +00:00
bf988dfc20
added queries for sample counts ( #2316 )
...
Co-authored-by: Alekhya <alekhya@Alekhyas-MBP.lan>
2021-09-07 14:05:09 -04:00
f7fff6351b
Streamline code to generate events_daily queries and re-run ( #2307 )
...
We're still checking the queries into the tree for now, but this moves the code to do so
into bqetl (more discoverable and easier to test) and generally streamlines things so it
will be easier to generate these queries at build-time. Also, check-in updated views from
a run of this code.
2021-08-31 18:56:46 +00:00
53e9a3172a
Remove amplitude views ( #2279 )
...
* Remove amplitude views
We never cleaned these up after removing the Amplitude export tasks in
https://github.com/mozilla/telemetry-airflow/pull/1328
* Update graveyard doc
Co-authored-by: Daniel Thorn <dthorn@mozilla.com>
2021-08-23 14:27:26 -04:00
b5b10ec5bb
remove latest_version column for histogram aggregates ( #2270 )
...
* remove latest_version column for histogram aggregates
* changed the run_glam_sql query based on comments
* remove latest_version dimension from histogram aggregates
2021-08-20 14:52:57 -04:00
4eb3dd1a60
Use ./script/bqetl in view scripts
2021-08-04 14:49:38 -07:00
aff449b54d
Revert "Revert "Rename cli.common back to cli.utils""
...
This reverts commit 8fcf0578ed
2021-08-04 14:49:38 -07:00
cb84f91d9f
Revert "Revert "Add view tests""
...
This reverts commit 63764c72cc
2021-08-04 14:49:38 -07:00
9c04bc8a13
Revert "Revert "bqetl view publish""
...
This reverts commit c4696d2e85
2021-08-04 14:49:38 -07:00
c4696d2e85
Revert "bqetl view publish"
...
This reverts commit d3a9f662e5
2021-07-29 10:47:24 -07:00
63764c72cc
Revert "Add view tests"
...
This reverts commit d0dfeb9701
2021-07-29 10:47:24 -07:00
8fcf0578ed
Revert "Rename cli.common back to cli.utils"
...
This reverts commit b0e4f9d644
2021-07-29 10:47:24 -07:00
b0e4f9d644
Rename cli.common back to cli.utils
2021-07-29 10:07:53 -07:00
d0dfeb9701
Add view tests
2021-07-29 10:07:53 -07:00
d3a9f662e5
bqetl view publish
2021-07-29 10:07:53 -07:00
d28ba56309
Remove generated-sql for glam ( #2215 )
2021-07-22 15:39:44 -07:00
cd85090c25
Fix #2212 - Write glam sql to directory specified by SQL_DIR ( #2213 )
...
* Add option to write to an alternative directory in generate_glam_sql
* Add alternative sql directory in glam portion of generate_sql
2021-07-22 14:13:59 -07:00
Daniel Thorn
Anna Scholtz
dependabot[bot]
akkomar
Jeff Klukas
Arkadiusz Komarzewski
Alekhya
Alexander Nicholson
whd
Anthony Miyaguchi
Will Lachance