mozilla
/
bigquery-etl
зеркало из https://github.com/mozilla/bigquery-etl.git
1
0
Форкнуть 0
Код Задачи Пакеты Проекты Релизы Вики Активность
4 227 коммитов 321 ветка 1 298 Теги 223 MiB
Граф коммитов

49 Коммитов

Автор SHA1 Сообщение Дата
Leli d98ae6bf2e
changing the formatting for documentation in cookbooks (#3970) 
* changing the formatting for documentation in cookbooks

* change to 1. 1. 1.
 2023-06-27 18:41:33 +02:00
Lucia 5dc9e405a8
Deng 1000 docs how to change control (#3903) 
* DENG-1000. Update docs with change control guide.

* DENG-1000. Update docs with change control guide.

* DENG-1000. Add sample PR to docs.

* DENG-1000. Add sample PR to docs.

* DENG-1000. Fix syntax of MD file.

---------

Co-authored-by: Lucia Vargas <lvargas@mozilla.com>
 2023-06-05 15:53:16 +00:00
Lucia dfef07ee40
DENG-1000. Update docs with change control guide. (#3901) 
* DENG-1000. Update docs with change control guide.

* DENG-1000. Update docs with change control guide.

* DENG-1000. Add sample PR to docs.

---------

Co-authored-by: Lucia Vargas <lvargas@mozilla.com>
 2023-06-05 14:53:20 +00:00
Daniel Thorn a0d810275b
Remove java dependency in favor of sqlglot (#3755) 2023-05-17 14:56:42 -07:00
Anna Scholtz 34f778c090
Stage deploy docs (#3648) 
* Docs for stage deploys

* Update cookbook for updating table schemas

* Docs for running stage deploys locally

* Update common workflow failures
 2023-03-10 09:10:32 -08:00
Alexander 60c85e7c54
Revert CI changes for private UDFs and add stub documentation - DENG-735 (#3652) 
* Revert "CI fixes for supporting private UDFs in bigquery-etl - DENG-735 (#3631)"

This reverts commit edcfe758f7.

* Added stub UDF for monetized_search

* Add docs for using a private internal UDF
 2023-03-10 11:41:46 -05:00
Leli 17c4fb35b1
add to documentation cookbook for testing (#3544) 2023-01-30 18:52:41 +01:00
akkomar 41fb32d7ef
Add a documentation note on faster single test execution (#3541) 2023-01-30 15:09:00 +01:00
Sean Rose d01ce2fc8a
Fix docs formatting (#3523) 
* Use `mdx_truly_sane_lists` to fix nested list formatting in the docs (by default, mkdocs requires nested lists to be indented by four spaces, which isn't how most nested lists in our docs have been formatted).
* Enable syntax highlighting for code blocks in the docs.
* Add support for GitHub Flavored Markdown task lists in the docs.
 2023-01-24 08:24:35 -08:00
Sean Rose 495ddcf39f
Correct BigQuery partitioning/clustering metadata in ETL `metadata.yaml` files (#3500) 
* Add missing BigQuery partitioning/clustering metadata.
* Correct existing BigQuery partitioning/clustering metadata.
* Allow partition `field` metadata field to be omitted.
* List partition `type` metadata field first.
 2023-01-12 13:58:53 -08:00
Frank Bertsch 0b6f1a6243
Update creating BQ dataset docs (#3487) 2023-01-06 08:23:23 -05:00
Anna Scholtz 5612836323 Automatically generate tables for Glean apps 2022-12-08 11:09:09 -08:00
Alexander f043631c9d
Update docs, Deploying clients_last_seen now requires skipping dryrun (#3405) 2022-12-06 09:07:59 -05:00
Lucia d92a732802
DENG-561 UDF to find earliest of a set of values (#3385) 
* UDF to retrieve the earliest not null value in an array.

* Test validated.

* Documentation upgrade for running the tests.

* Update sql/mozfun/norm/get_earliest_value/metadata.yaml

Co-authored-by: Frank Bertsch <frank.bertsch@gmail.com>

* Update documentation.

Co-authored-by: Lucia Vargas <lvargas@mozilla.com>
Co-authored-by: Frank Bertsch <frank.bertsch@gmail.com>
 2022-12-01 23:22:18 +01:00
Daniel Thorn c281400486
Enforce isort via pytest (#3384) 2022-11-30 11:45:05 -08:00
Winnie Chan 2f9119c056
Updated file paths (#3305) 2022-10-25 14:26:47 -07:00
Anna Scholtz bc1ceb16be Add runbook for adding new Glean apps 2022-10-25 09:15:09 -07:00
Frank Bertsch 00a04b699c
Move to python 3.10 (#3236) 
* WIP: Move to python 3.10

* Remove netlify
 2022-09-22 15:39:40 -04:00
Rebecca BurWei 59cd108a78
docs for data scientists (#3192) 
* docs

* Update docs/cookbooks/common_workflows.md

Co-authored-by: Anna Scholtz <anna@scholtzan.net>

* Update docs/cookbooks/common_workflows.md

Co-authored-by: Anna Scholtz <anna@scholtzan.net>

Co-authored-by: Anna Scholtz <anna@scholtzan.net>
 2022-08-31 11:45:06 -05:00
Rebecca BurWei 98def555fd
Add workflow docs for data scientists (#3179) 
* Update common_workflows.md

* Update docs/cookbooks/common_workflows.md

Co-authored-by: Anna Scholtz <anna@scholtzan.net>

Co-authored-by: Anna Scholtz <anna@scholtzan.net>
 2022-08-30 11:14:05 -05:00
Anna Scholtz 81fbdf6c01 Move doc validation and generation commands to CLI 2022-08-25 08:51:41 -07:00
Alexander 588d468dc8
Hoist schemas in SQL tests up to table dir (#3145) 2022-08-17 13:11:24 -04:00
Lucia dfef0030cf
Update docs to remove the default mozilla-public-data as default in docs (#2960) 
* Update docs to remove the default instruction to use  `mozilla-public-data` as the project to create a new table. Clarify to specify the project.

* Update documentation to create a derived dataset. Replace the use of init files for the use of schemas.

* Small corrections

* Small corrections

Co-authored-by: Lucia Vargas <lvargas@mozilla.com>
 2022-05-23 11:23:17 +02:00
Lucia da36b8b3f8
Update bigquery-etl cookbooks (#2935) 
* Ignore .idea/ folder in pycharm

* Make instructions to add field generic with example. Add recommendations to delete field.

* Reformat numbered list for consistency.

* Update instructions based on PR feedback.

Co-authored-by: Lucia Vargas <lvargas@mozilla.com>
 2022-05-04 11:10:34 +02:00
Sean Rose 3fb6ebc160
Specify pyjnius in `requirements.in` (#2924) 2022-04-29 12:23:47 -07:00
Daniel Thorn 0235d6087c
Reapply "Update to zetasql 2022.02.1 (#2862)" (#2881) 2022-04-07 14:02:20 -07:00
Daniel Thorn 22835e5706
Revert "Update to zetasql 2022.02.1 (#2862)" (#2879) 
This reverts commit 9c9761feac.
 2022-04-07 12:03:09 -07:00
Daniel Thorn 9c9761feac
Update to zetasql 2022.02.1 (#2862) 2022-04-05 18:55:52 +02:00
Daniel Thorn 4406437be2
Use --no-deps when installing compiled requirements files (#2752) 2022-02-24 21:36:47 +00:00
Jeff Klukas 8e125448c7
Bug 1727081 Add link to Data Access Workgroups docs (#2623) 
See https://bugzilla.mozilla.org/show_bug.cgi?id=1727081
 2022-01-05 19:04:37 +00:00
kik-kik 8898f3870f Apply suggestions from code review by @wlach 
Thank you @wlach for the review and suggestions 🙇

Co-authored-by: Will Lachance <wlachance@mozilla.com>
 2021-12-16 15:13:27 +01:00
= 9a868a61ea created a new doc for explaining Airflow tags 2021-12-16 15:13:27 +01:00
= c2f1d137d1 updated cookbooks for derived_dataset to include tag changes 2021-12-16 15:13:27 +01:00
Will Lachance 684c0c7930
Minor fixes to the build docs workflow formatting (#2577) 2021-12-14 15:14:46 -05:00
Alexander Nicholson 9aacb3bff7
Added instructions on building and serving docs (#2560) 
Co-authored-by: Will Lachance <wlachance@mozilla.com>
 2021-12-09 10:21:24 -05:00
Will Lachance 1d6fea8f2d
Big docs/README refactoring (#2527) 
* Move most reference and cookbook documentation in README.md into
  the "reference" and "cookbook" sections of the generated documentation,
  respectively.
* Try to steer people to the generated docs inside the README.md (since
  it is now basically just a set of quickstart instructions)
* Provide a bit of guidance that this repository isn't great for 3rd
  party contributors in a new CONTRIBUTING.md.

Co-authored-by: Jeff Klukas <jklukas@mozilla.com>
 2021-11-29 21:28:07 +00:00
Will Lachance dde5278671
Specify that --project-id is required for bqetl backfill (#2396) 2021-09-29 19:12:46 -04:00
Jeff Klukas 860b7e7367
Bug 1712828 Add urlbar_picked keyed scalars to clients_daily (#2115) 
* Bug 1712828 Add urlbar_picked keyed scalars to clients_daily

Addresses https://bugzilla.mozilla.org/show_bug.cgi?id=1712828
and also adds the preferences_paneHome keyed scalar from
https://bugzilla.mozilla.org/show_bug.cgi?id=1715377

* Update schemas

* Update workflow docs
 2021-06-14 09:11:41 -04:00
Anna Scholtz 0788f76fe4 Update docs for adding clients_daily fields 2021-05-19 12:51:11 -07:00
Anna Scholtz f18d19fd5f Update workflow docs 2021-05-07 10:30:02 -07:00
Jeff Klukas beaf124c29 Add clients_last_seen_joined to schema update workflow docs 2021-05-06 10:10:26 -04:00
Jeff Klukas 2c8f455162
Bug 1709595 Add new attribution experiment fields to clients_daily (#2004) 
* Implement schema.yaml for the clients_daily chain

* Bug 1709595 Add new attribution experiment fields to clients_daily

* Add schema for clients_daily_joined_v1

* yamllint

* DAG update

* Update tests

* Add --force option
 2021-05-05 15:17:13 -04:00
whd 7c1b03934b
Default branch (#1939) 
* Rename default branch

* Rename branch

* Update circleci for default branch name
 2021-04-06 21:15:21 +00:00
Anna Scholtz be1a0c4ec9 Workflow docs, mark schema generation step as optional 2021-02-25 13:58:57 -08:00
Anna Scholtz 6142b1562e Common workflow docs link to scheduled query tutorial; docs refactoring 2021-02-25 13:58:57 -08:00
Anna Scholtz f2a48a180d Add common workflow descriptions 2021-02-25 13:58:57 -08:00
Anna Scholtz d9ed547a98 Docs for common workflows: query scheduling 2021-02-25 13:58:57 -08:00
Anna Scholtz d5fe8fd28b
Set mozilla-public-data project in mozregression_aggregates_v1 init.sql (#1837) 2021-02-24 14:40:02 -05:00
William Lachance 5148a7f775
New cookbook on creating a derived dataset (#1781) 
Co-authored-by: Anna Scholtz <anna@scholtzan.net>
Co-authored-by: Jeff Klukas <jklukas@mozilla.com>
 2021-02-11 12:11:27 -05:00