03357769cc
Move view, schema and remaining configs to bqetl_project.yaml ( #4051 )
...
* Move view configs to bqetl_project.yaml
* Move schema config to bqetl_project.yaml
* Move docs config to bqetl_project.yaml
* Replace remaining configs
2023-07-11 13:10:57 -07:00
3286508bc5
Update view metadata in a single update_table() operation ( #4017 )
2023-07-04 10:55:55 -07:00
7c90d5f8e7
Publish view metadata ( #3909 )
2023-06-29 16:28:17 -04:00
46397f42b5
Deploy UDF references to stage for views ( #3849 )
2023-05-23 13:10:40 -07:00
08b45a40fe
Jinja queries support ( #3691 )
...
* Support Jinja templating in query files
* Formatting for Jinja
* ./bqetl query render command
* Fix running templates
2023-03-30 11:00:12 -07:00
51def19185
Bug 1825545 - Revert "Support Jinja templating in query files ( #3685 )" ( #3689 )
...
Bug 1825545 - This reverts commit a1c51124ec
2023-03-30 10:47:17 -04:00
a1c51124ec
Support Jinja templating in query files ( #3685 )
...
* Support Jinja templating in query files
* Formatting for Jinja
* ./bqetl query render command
2023-03-29 10:38:08 -07:00
06d7baa367
Skip client_deduplication view validation ( #3645 )
2023-03-08 13:23:01 -08:00
22e54ccb5f
[Bug 1812301] Publish only string typed labels ( #3530 )
...
* [Bug 1812301] Publish only string typed labels
* Document label publishing
2023-01-26 09:24:49 -08:00
6fb2c91a1d
Trim semicolons when comparing view sql ( #3439 )
2022-12-08 15:06:04 -08:00
f1699b21ed
Add `bqetl views clean` command ( #3433 )
...
to remove managed views from BigQuery when they are removed from sql dir
2022-12-08 14:48:03 -08:00
c807de71c8
Add missing condition for target project in view.has_changes ( #3437 )
2022-12-08 14:03:22 -08:00
e40f43bfed
Use target_view not view_identifier when updating labels ( #3431 )
2022-12-08 11:19:23 -08:00
42d48be752
Rename views directories to match convention ( #3413 )
2022-12-08 10:51:34 -08:00
9e52d19909
Ignore missing view schema files ( #3429 )
2022-12-08 10:27:16 -08:00
cf6ce002d3
Publish views serially and default to only publishing changes ( #3424 )
2022-12-08 10:17:01 -08:00
52be263f2e
Add option to skip publishing authorized views ( #3418 )
2022-12-07 10:53:46 -08:00
ddbabeef05
Publish views in order by dependencies ( #3411 )
2022-12-06 17:16:15 -08:00
c281400486
Enforce isort via pytest ( #3384 )
2022-11-30 11:45:05 -08:00
ebc6c0d0ab
Move list_broken_views to CLI
2022-08-24 09:01:29 -07:00
9a5a98779a
Remove scripts that have been replaced by CLI
2022-07-19 08:08:49 -07:00
f6da873783
Move DAG generation to CI
2022-01-07 14:25:04 -08:00
89e0da4f2b
Move stable view generation to sql_generators
2022-01-05 12:26:52 -08:00
f6a9b4cf01
Simplify experiment_enrollment_cumulative_population_estimate query
2021-11-19 11:48:29 -08:00
33f7c84202
add glam_etl to non user facing view ( #2476 )
2021-11-04 13:01:34 -04:00
52b5eb678e
Speed up stable view generation
2021-10-22 09:18:44 -07:00
86c7779595
Avoid error messages in generate_stable_views ( #2284 )
...
As of https://github.com/mozilla/bigquery-etl/pull/2118 we see error messages
from any `bqetl` command that calls `generate_stable_views` like:
```
Cannot get Bigquery schema for mozilla-pipeline-schemas-9e81d822064c5a8336d3f2e61cc67cdf58d4a010/schemas/glean/glean/glean.1.schema.json: "filename 'mozilla-pipeline-schemas-9e81d822064c5a8336d3f2e61cc67cdf58d4a010/schemas/glean/glean/glean.1.bq' not found"
Cannot get Bigquery schema for mozilla-pipeline-schemas-9e81d822064c5a8336d3f2e61cc67cdf58d4a010/schemas/metadata/credentials/credentials.1.schema.json: "filename 'mozilla-pipeline-schemas-9e81d822064c5a8336d3f2e61cc67cdf58d4a010/schemas/metadata/credentials/credentials.1.bq' not found"
...
```
These correspond to schema "fragments" that have no concrete BQ tables.
We need to check for the existence of `mozPipelineMetadata` _before_
trying to access a relevant BQ schema in order to avoid these.
2021-08-23 14:07:36 -04:00
a912f6ca4c
Fix skipping view validations when publishing
2021-08-06 11:44:38 -07:00
ffd5a1eaf2
Fix typo in comment ( #2249 )
2021-08-06 17:08:34 +00:00
3f413145da
Avoid checking view references when publishing views
2021-08-06 09:46:59 -07:00
218ed63368
Fix skipping publishing and validation of views
2021-08-04 16:41:23 -07:00
cb84f91d9f
Revert "Revert "Add view tests""
...
This reverts commit 63764c72cc
2021-08-04 14:49:38 -07:00
468db4b484
Revert "Revert "bqetl view create""
...
This reverts commit e8aa2c369b
2021-08-04 14:49:38 -07:00
9c04bc8a13
Revert "Revert "bqetl view publish""
...
This reverts commit c4696d2e85
2021-08-04 14:49:38 -07:00
858919d350
Revert "Revert "Add bqetl view validate command""
...
This reverts commit a0ccd7602b
2021-08-04 14:49:38 -07:00
a0ccd7602b
Revert "Add bqetl view validate command"
...
This reverts commit 62c85cb36c
2021-07-29 10:47:24 -07:00
c4696d2e85
Revert "bqetl view publish"
...
This reverts commit d3a9f662e5
2021-07-29 10:47:24 -07:00
e8aa2c369b
Revert "bqetl view create"
...
This reverts commit 410fa3fc7d
2021-07-29 10:47:24 -07:00
63764c72cc
Revert "Add view tests"
...
This reverts commit d0dfeb9701
2021-07-29 10:47:24 -07:00
d0dfeb9701
Add view tests
2021-07-29 10:07:53 -07:00
410fa3fc7d
bqetl view create
2021-07-29 10:07:53 -07:00
d3a9f662e5
bqetl view publish
2021-07-29 10:07:53 -07:00
62c85cb36c
Add bqetl view validate command
2021-07-29 10:07:53 -07:00
dfb2020eda
Prefix dataset with project id
2021-06-15 08:54:39 -04:00
143c27a353
Don't validate funnel view
2021-06-14 21:42:37 -04:00
03d4b1d39f
Handle exceptions and float type when adding descriptions to stable views
2021-06-14 10:38:58 -07:00
8809da61d7
Add field descriptions to stable views
2021-06-14 10:38:58 -07:00
7851b5d37a
Bug 1708264 - Generate dataset metadata ( #1988 )
...
* Generate dataset_metadata.yaml based on doctypes
* Exclude pioneer from dataset_metadata.yaml creation
* Mark `static` as user-facing
* Use namespace in friendly names, and use default descriptions
2021-05-03 11:47:43 -04:00
2cab2338e9
Remove fxa_amplitude_email_clicks ( #1892 )
...
* Remove fxa_amplitude_email_clicks
Per @cvalaas, the SFMC email system is now retired, so the underlying table
here is no longer receiving updates.
Depends on https://github.com/mozilla/telemetry-airflow/pull/1268
* Add diff-filter
* yamllint hint
* More yamllint fixing
2021-03-15 14:02:20 -04:00
789f476cd2
Move materialized view into init.sql files
2021-03-03 15:46:45 -08:00
Anna Scholtz
Linh Nguyen
kik-kik
Daniel Thorn
whd
Alekhya
Jeff Klukas
Frank Bertsch