Граф коммитов

10 Коммитов

Автор SHA1 Сообщение Дата
Curtis Morales 247729d217
In validation, only read schema files when necessary (#6252)
* Remove deleted table from skip list

* Parse schema file in validate_shredder_mitigation function so it works on priv-bqetl

* Parse schema file in validate_shredder_mitigation function so it works on priv-bqetl

* Clean up test
2024-09-24 17:03:28 +00:00
Lucia 074731db7b
CI validation of tables with the shredder_mitigation label (#6217)
* Larger wildcards to reduce the chance of collision with actual values.

* Formatting

* Update bigquery_etl/metadata/validate_metadata.py

Co-authored-by: Ben Wu <12437227+BenWu@users.noreply.github.com>

* Add test for validate_metadata.validate, add profile_id and profile_group_id to id-level_columns file.

* Update tests/cli/test_cli_metadata.py

Co-authored-by: Ben Wu <12437227+BenWu@users.noreply.github.com>

---------

Co-authored-by: Ben Wu <12437227+BenWu@users.noreply.github.com>
2024-09-18 18:44:06 +00:00
Anna Scholtz 7260510cc3
Add monitoring metadata support (#6152)
Co-authored-by: Alekhya <88394696+alekhyamoz@users.noreply.github.com>
2024-09-04 13:03:59 -04:00
Winnie Chan 33f9017c75
DENG-2823: Added deprecate cli command (#5219)
* Added deprecate cli command

* Fixed typo

* Fixed failed tests

* Fixed deletion date label

* Update bigquery_etl/metadata/parse_metadata.py

Co-authored-by: Sean Rose <1994030+sean-rose@users.noreply.github.com>

* Fixed deletion date

* Fixed arguments optional

* Added return back

* Added invalid deletion date test

---------

Co-authored-by: Sean Rose <1994030+sean-rose@users.noreply.github.com>
2024-03-19 11:17:32 -07:00
Winnie Chan 6c501a620c
Deng-2845: Remove default deprecated false (#5118)
* Removed deprecated field

* Removed deprecated false in metadata yaml

* Fixed test
2024-02-27 10:33:50 -08:00
Winnie Chan 8ec7516157
Issue 4135: Added publish metadata cli command (#5011)
* Added publish metadata cli command

* Removed publish metadata script
2024-02-12 11:12:14 -08:00
Anna Scholtz b0a1a32246
Speed up generate-sql (#4921)
* Speed up glean_usage generator

* Refactoring
2024-01-31 12:08:29 -08:00
Anna Scholtz 073b1f050d
Fix dataset deprecation metadata (#4874)
* Update dataset workgroup_access when deprecated: true

* Update deprecation metadata tests

* Add metadata.yaml files in telemetry_derived for tables that are managed through other tooling

* Deprecate telemetry_derived datasets
2024-01-26 22:03:54 +00:00
Alekhya 2e916eb856
DENG1381 - Add bqetl support for deprecation metadata (#4213)
* Support bq dataset deprecation process (metadata)

* Add bqetl metadata cli command

* Initial draft for adding deprecation support to bqetl

* Incorporate Anna's feedback

* Fix based on whd's feedback

* Fix ci issues

* Remove unnecessary logic from metadata.py

* Add dataset metadata yaml for ga_derived

* Ignore dirs that do not have dataset_metadata yaml

* Remove unwanted dataset metadata yamls

* Update bigquery_etl/cli/metadata.py

Co-authored-by: whd <whd@users.noreply.github.com>

---------

Co-authored-by: whd <whd@users.noreply.github.com>
2023-09-12 18:47:54 +00:00
Lucia c34653778f
DENG-774 Change Control for active_users_aggregates (#3687)
* DENG-774 Add change control to active_users_aggregates and test.

* DENG-774 Add test coverage.
---------

Co-authored-by: Lucia Vargas <lvargas@mozilla.com>
2023-04-14 15:49:19 +02:00