Граф коммитов

38 Коммитов

Автор SHA1 Сообщение Дата
Alexander fbf9894251
docs: DENG-809 add instructions for managed backfills (#5377)
* docs: DENG-809 add instructions for managed backfills

* Apply suggestions from code review

Co-authored-by: Winnie Chan <10429026+wwyc@users.noreply.github.com>

* Additional doc updates

---------

Co-authored-by: Winnie Chan <10429026+wwyc@users.noreply.github.com>
2024-04-17 13:13:53 -04:00
Anna Scholtz eac0ac80c2
Remove telemetry_derived init.sql files (#5342)
* Remove init.sql files for telemetry_derived queries

* Remove init.sql for events_daily

* Remove init.sql from skip lists

* Remove init.sql references from tooling

* Add schema for baseline_clients_first_seen
2024-04-10 15:36:30 -07:00
Sean Rose 25e4c224a0
Support waiting for tables/partitions to exist before running ETLs (#5039)
* Implement `depends_on_tables_existing` and `depends_on_table_partitions_existing` scheduling metadata.

* Refactor repeated timedelta validation logic into `validate_timedelta_string` utility function.

* Replace Google Search Console empty-check ETLs with `depends_on_table_partitions`.
2024-02-15 14:38:52 -08:00
Anna Scholtz b0387fb7de
Remove referenced_tables usages (#4834)
* Remove referenced_tables usages

* Resolve check dependencies when generating DAGs

* Add tests for checks automatically resolving dependencies

* Incorporate feedback for removing referenced_tables

* Use depends_on for empty_checks

* More depends_on and fixes
2024-01-22 12:41:05 -08:00
kik-kik 85216b9c2a
added an option to matches_pattern to be able to specify percentage threshold / tolerance for failures for the check (#4716) 2023-12-15 15:44:49 +01:00
Leli 699b325a0e
add accepted values check (#4650)
* add accepted values check

* added semicolon
2023-12-15 13:34:04 +01:00
kik-kik 34d6a463dc
feat(DENG-2175): added matches_pattern etl checks macro and updated tests to use it (#4683)
* added matches_pattern etl checks macro and updated tests to use it

* added matches_pattern etl check macro to the data checks docs
2023-12-13 16:10:34 +01:00
kik-kik 2db2f3e2de
feat(DENG-2156): added value_length check and updated some of the ETL checks to use the macro (#4672)
* added value_length check and updated some of the ETL checks to use the macro

* added the new check macro to the data checks docs

* implemented lelilia feedback from PR#4672

* simplified the sql logic for the value_length check
2023-12-11 16:31:25 +01:00
Mike Williams 191f935d97
add triage/confidential tag to docs (#4678) 2023-12-11 09:56:38 -05:00
Jan-Erik Rediger a26eaf5e59 Docs: Link to section on the same page
The separate chapter was removed in #4293
2023-12-06 12:20:57 +01:00
Leli f84a097524
Feat/deng 2046/migrating telemetry derived active users aggregates v1 dim checks to etl checks (#4641)
* Migrated DIM checks over to ETL checks for telemetry_derived.active_users_aggregates_v1

* rewrite

* code review suggestions

* add doc

* rename

---------

Co-authored-by: kik-kik <kignasiak@mozilla.com>
2023-12-04 17:19:48 +01:00
Leli 9c1d19fc0b
change docs (#4629) 2023-11-30 17:03:54 +01:00
Anna Scholtz dffbb3afef
Remove generated DAGs from main (#4507) 2023-11-01 15:05:59 -07:00
kik-kik 42bfa1409e
Minor tweaks to the data checks docs (#4309) 2023-09-20 10:57:39 +02:00
Alekhya 01ff67a29f
Rename min_rows check macro to min_row_count (#4301) 2023-09-14 15:06:41 -04:00
Anna Scholtz 9a6a2df03b
Updating and merging data check docs (#4293) 2023-09-13 14:48:47 -07:00
kik-kik 7afc4c44f1
docs(DENG-960): bqetl data checks cli docs (#4200)
* Small tweaks made to the cli cmds comments / help display for data checks

* added usage docs to data_checks reference docs

* Apply suggestions from code review provided by scholtzan

Co-authored-by: Anna Scholtz <anna@scholtzan.net>

---------

Co-authored-by: Anna Scholtz <anna@scholtzan.net>
2023-08-24 17:24:23 +02:00
kik-kik de35707b1d
docs(DENG-961): Added reference doc for data checks and a cookbook for adding new data checks (#4162)
* added reference doc for data checks and a cookbook for adding new data checks

* Apply suggestions from code review by ascholtz

Co-authored-by: Anna Scholtz <anna@scholtzan.net>

---------

Co-authored-by: Anna Scholtz <anna@scholtzan.net>
2023-08-09 17:12:55 +02:00
Anna Scholtz d5a6dc97f4
Docs for new `bqetl_project.yaml` (#4018)
* Add ConfigLoader and move dry run skip to bqetl_project.yaml

* Update bqetl configuration docs
2023-07-10 09:50:22 -07:00
Curtis Morales b7b1b835ba
Add trigger_rule as an option for generated airflow tasks (#3772)
* Add trigger_rule as an option for generated airflow tasks

* Add test

* Move trigger rule options to enum and add to documentation
2023-06-27 13:58:52 -04:00
Sean Rose 45836abc69
Fix metrics docs formatting (#3704)
* Remove trailing whitespace to satisfy pre-commit check.

* Remove blank line that's breaking docs formatting.

* Re-add blank lines lost in copy+paste.
2023-04-05 08:07:13 -07:00
Anna Scholtz 48d8c7603d
Metric hub integration - rewrite SSL ratios to use metrics (#3698)
* Add metrics.data_source()

* Rewrite SSL ratios to use metrics

* Fix docs formatting
2023-04-04 15:41:44 -07:00
Anna Scholtz eb5d63a8f5
Metric-hub doc tweaks (#3700) 2023-04-04 10:18:06 -07:00
Anna Scholtz d737925da1
Metric hub integration docs (#3697)
* Add metrics.data_source()

* Docs for referencing metrics

* Add docs for using metrics in queries
2023-04-04 09:56:47 -07:00
Anna Scholtz 34f778c090
Stage deploy docs (#3648)
* Docs for stage deploys

* Update cookbook for updating table schemas

* Docs for running stage deploys locally

* Update common workflow failures
2023-03-10 09:10:32 -08:00
kik-kik 414a3c5d77
Fixed typo in airflow_tags.md (#3589)
* Fixed typo in airflow_tags.md

As Sean pointed out, the tag provided in the docs was incorrect. This change fixes that.

* Apply suggestions from code review

Co-authored-by: Sean Rose <1994030+sean-rose@users.noreply.github.com>

---------

Co-authored-by: Sean Rose <1994030+sean-rose@users.noreply.github.com>
2023-02-16 11:10:58 +01:00
kik-kik 67ac75a04b
:dded description for triage/record_only tag (#3576) 2023-02-10 10:00:31 -05:00
Anna Scholtz 22e54ccb5f
[Bug 1812301] Publish only string typed labels (#3530)
* [Bug 1812301] Publish only string typed labels

* Document label publishing
2023-01-26 09:24:49 -08:00
Sean Rose d01ce2fc8a
Fix docs formatting (#3523)
* Use `mdx_truly_sane_lists` to fix nested list formatting in the docs (by default, mkdocs requires nested lists to be indented by four spaces, which isn't how most nested lists in our docs have been formatted).
* Enable syntax highlighting for code blocks in the docs.
* Add support for GitHub Flavored Markdown task lists in the docs.
2023-01-24 08:24:35 -08:00
Sean Rose afea1c35b9
Remove dependencies on the `mozdata` project from ETL (#3496)
Having dependencies on things in `mozdata` can cause issues with deployments, as deploying things to `mozdata` is usually a separate secondary step.
2023-01-23 08:59:04 -08:00
Lucia af64945584
DENG-127 Update docs for backfills (#3150)
* DENG-127 Update docs to include that backfills are documented in Bugzilla.

* DENG-127 Add query for backfill validation.

* DENG-127 Add query for backfill validation.

Co-authored-by: Lucia Vargas <lvargas@mozilla.com>
2022-08-18 18:22:16 +02:00
Anna Scholtz 9a5a98779a Remove scripts that have been replaced by CLI 2022-07-19 08:08:49 -07:00
Anna Scholtz e7d1e1243c Add task marker docs 2022-06-22 11:05:25 -07:00
kik-kik de2c7ca1dd
removed link to Google Docs as public should not try to access it (#3000) 2022-06-01 14:45:37 +00:00
Anna Scholtz b1a1f5a484 Add fivetran task docs 2022-01-11 11:23:32 -08:00
kik-kik 8898f3870f Apply suggestions from code review by @wlach
Thank you @wlach for the review and suggestions 🙇

Co-authored-by: Will Lachance <wlachance@mozilla.com>
2021-12-16 15:13:27 +01:00
= 9a868a61ea created a new doc for explaining Airflow tags 2021-12-16 15:13:27 +01:00
Will Lachance 1d6fea8f2d
Big docs/README refactoring (#2527)
* Move most reference and cookbook documentation in README.md into
  the "reference" and "cookbook" sections of the generated documentation,
  respectively.
* Try to steer people to the generated docs inside the README.md (since
  it is now basically just a set of quickstart instructions)
* Provide a bit of guidance that this repository isn't great for 3rd
  party contributors in a new CONTRIBUTING.md.

Co-authored-by: Jeff Klukas <jklukas@mozilla.com>
2021-11-29 21:28:07 +00:00