fbf9894251
docs: DENG-809 add instructions for managed backfills ( #5377 )
...
* docs: DENG-809 add instructions for managed backfills
* Apply suggestions from code review
Co-authored-by: Winnie Chan <10429026+wwyc@users.noreply.github.com>
* Additional doc updates
---------
Co-authored-by: Winnie Chan <10429026+wwyc@users.noreply.github.com>
2024-04-17 13:13:53 -04:00
84ee88e2b9
Dependabot/pip/black 24.1.1 fix ( #5027 )
...
* Bump black from 23.10.1 to 24.1.1
Bumps [black](https://github.com/psf/black ) from 23.10.1 to 24.1.1.
- [Release notes](https://github.com/psf/black/releases )
- [Changelog](https://github.com/psf/black/blob/main/CHANGES.md )
- [Commits](https://github.com/psf/black/compare/23.10.1...24.1.1 )
---
updated-dependencies:
- dependency-name: black
dependency-type: direct:production
update-type: version-update:semver-major
...
Signed-off-by: dependabot[bot] <support@github.com>
* Reformat files with black to fix dependabot update.
* Reformat with black 24.1.1. Update test dag with required space.
* Update test dags.
---------
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-02-19 15:27:34 +01:00
a70b2aa689
Support symlinks ( #4881 )
...
* Avoid using `Path.glob()` or `Path.rglob()` for recursive file searches.
Because they don't currently support following symlinks (they will in Python 3.13).
* Specify `followlinks=True` as necessary when calling `os.walk()`.
2024-01-24 13:02:43 -08:00
e0996c20cd
Bqetl on rich-cli ✨ 💸 ( #3775 )
2024-01-03 11:30:54 -05:00
7a80984757
DENG-1193 Deprecate generated dataset docs ( #4657 )
2023-12-15 12:17:26 -05:00
463dc15bf1
Support shared-prod UDFs ( #4708 )
2023-12-14 13:45:13 -05:00
5c0748cf79
Add missing / for generating docs ( #4055 )
2023-07-11 15:55:01 -07:00
03357769cc
Move view, schema and remaining configs to bqetl_project.yaml ( #4051 )
...
* Move view configs to bqetl_project.yaml
* Move schema config to bqetl_project.yaml
* Move docs config to bqetl_project.yaml
* Replace remaining configs
2023-07-11 13:10:57 -07:00
ca2c84e9b5
Add to bqetl docs on query run ( #3879 )
...
* change examples for bqetl query run and remove weird indentation from code examples
* actually upload the correct file
* note on order of parameters
2023-06-22 13:52:45 +00:00
f17ba7ad25
add input and output parameters to udf and stored procedure documenattion ( #3843 )
...
* rename udf_functions to mozfun_doc_functions and add edgecases
* refactor generate_mozfun_docs
* add parameters to stored procedures
* bolden input and output
2023-05-23 18:20:15 +02:00
bba1758932
fix udf parsing for docs ( #3821 )
2023-05-22 13:49:31 -04:00
10f9994901
fix #3756 : parse inputs and outputs from UDFs for docs ( #3778 )
...
* formatting
* refactor parsing
* tweaks to udf parser utils for edge cases
* refactor UDF input/output parser to use utils; formatting
* add check for edge case in output regex
Co-authored-by: Leli <33942105+lelilia@users.noreply.github.com>
* fix test for udf output util
* remove inconsistent formatting of inputs/outputs
---------
Co-authored-by: Lucia <30448600+lucia-vargas-a@users.noreply.github.com>
Co-authored-by: Leli <33942105+lelilia@users.noreply.github.com>
2023-05-22 12:45:29 -04:00
81fbdf6c01
Move doc validation and generation commands to CLI
2022-08-25 08:51:41 -07:00
7998cafaab
Bump click from 8.0.4 to 8.1.0 ( #2838 )
2022-03-29 20:46:31 +02:00
c1925039de
Extract mozfun docs generation and modify call order ( #2689 )
2022-01-28 11:44:17 -05:00
f6da873783
Move DAG generation to CI
2022-01-07 14:25:04 -08:00
f04ee81d65
Added schemas to derived dataset docs using schema files ( #2614 )
2022-01-05 16:57:53 -05:00
d6936dfb8d
quick fix for null metadata descriptions ( #2585 )
2021-12-16 16:50:36 -05:00
73d6256799
Support dataset READMEs and metadata, cosmetic changes ( #2579 )
2021-12-16 14:28:34 -05:00
d2ebd03755
Derived dataset docs generation - refactor and support READMEs ( #2559 )
...
* Refactor derived dataset generation script - more pathlib functions
* Support View READMEs in generated docs
2021-12-08 19:31:40 -05:00
f26651463f
Use base url when linking to cookbook ( fixes #2402 ) ( #2425 )
2021-10-12 12:01:48 -04:00
0f898fc501
Fix not enough values to unpack for referenced tables
2021-04-06 15:09:14 -07:00
0fb24f1c1f
Use zetasql for doc generation
2021-03-31 09:04:23 -07:00
9fbbac8ee9
Fix referenced_tables error in generate_docs
2021-03-30 11:02:38 -07:00
f2b920d085
Add info comment for \b markers
2021-03-25 14:16:06 -07:00
e6478806a8
Use jinja macros for recursive bqetl docs generation
2021-03-25 14:16:06 -07:00
1554d045cb
Add docs for bqetl routine and mozfun commands
2021-03-25 14:16:06 -07:00
5fd43bcd42
Add examples for bqetl queries, format and dryrun
2021-03-25 14:16:06 -07:00
d0dd7e0dce
Add options and arguments to bqetl docs
2021-03-25 14:16:06 -07:00
de2c7c9945
Generate bqetl command docs from click command help text
2021-03-25 14:16:06 -07:00
9b1d409185
Fix use of parse_routine.sub_local_routines in validate-docs ( #1860 )
2021-03-01 15:03:29 -08:00
a190e18264
Automatically sort python imports ( #1840 )
2021-02-24 17:11:52 -05:00
34cb4d559a
Copy `generated-sql` docs content to `sql` ( #1835 )
2021-02-24 15:31:31 -05:00
2e43803cf1
Strip temporary dir name from filepath for docs generation ( #1819 )
...
* Remove "tmp/generated-sql" from dir name
Since docs are generated from 'tmp/generated-sql' dir,
we need to remove 'tmp/generated-sql' from file names
so they can be checked against the SKIP list
* Defide more specific destination path for mozfun
* Add pages sorting to mozfun
2021-02-23 09:27:22 -08:00
28f15e16e5
Use generated SQL content as source for docs ( #1811 )
2021-02-16 13:33:53 -08:00
4f26049318
Generate referenced tables for views ( #1763 )
...
* WIP
* Get referenced tables for mozdata
* Address review comments and add tests
* Try 'SELECT from {view}' method
This approach returns a lot of unfound table errors,
it seems like there's not a direct 1:1 correlation
between bigquery-etl table names and BigQuery datasets
on the cloud
* Add tests
* Lint dryrun.py
* Add some minor cleanups
* Add test for multiple referenced tables
* Sort referenced table results
2021-02-16 11:49:19 -08:00
1ea5edbb95
Fix table-level links ( #1804 )
...
(fixes #1801 )
2021-02-12 13:52:48 -08:00
5148a7f775
New cookbook on creating a derived dataset ( #1781 )
...
Co-authored-by: Anna Scholtz <anna@scholtzan.net>
Co-authored-by: Jeff Klukas <jklukas@mozilla.com>
2021-02-11 12:11:27 -05:00
350291bd1a
Only show available urls in source ( #1762 )
...
Co-authored-by: Anthony Miyaguchi <amiyaguchi@mozilla.com>
2021-02-05 12:22:17 -08:00
dd10f7a7f7
Generate summaries of derived tables ( fixes #1589 ) ( #1735 )
...
* Generate summaries of derived tables
* Generate mozdata docs
* Address requested changes
2021-02-04 13:05:26 -08:00
0bb87d79f4
Update docs/index.md ( #1590 ) ( #1625 )
...
* Update docs/index.md
Co-authored-by: Anna Scholtz <anna@scholtzan.net>
2020-12-15 15:53:19 -08:00
be961f10a0
Add links to metadata.yaml editor and source dictory ( #1242 ) ( #1618 )
...
* Add links to edit and source directory
* Update docs/mkdocs.yml
Co-authored-by: Anna Scholtz <anna@scholtzan.net>
Co-authored-by: jailang <lynjailang@gmail.com>
Co-authored-by: Anna Scholtz <anna@scholtzan.net>
2020-12-15 11:52:46 -08:00
b0aec90544
Fix UDF tests
2020-10-08 10:30:22 -07:00
a84838466e
Rename routine tests
2020-10-08 10:30:22 -07:00
67c5265b6f
Rename udf module to routine
2020-10-08 10:30:22 -07:00
22d8798ef0
Rename functions that process UDFs and stored procedures to routine
2020-10-08 10:30:22 -07:00
56c922a494
Fix generated docs path
2020-10-06 09:37:11 -07:00
8a68128ee7
Clarify which functions are procedures in docs
2020-10-06 10:54:25 -04:00
f131c585a5
Rename DEFAULT_PROJECTS to DEFAULT_PROJECTS_DIRS
2020-10-05 12:59:58 -07:00
a16bc2824f
Fix validating docs
2020-10-05 12:59:58 -07:00
Alexander
Lucia
Sean Rose
Anna Scholtz
Leli
Mike Williams
dependabot[bot]
Alexander Nicholson
Will Lachance
Daniel Thorn
Linh Nguyen
Linh Nguyen
Frank Bertsch