Граф коммитов

9 Коммитов

Автор SHA1 Сообщение Дата
Anna Scholtz 7024b6976e
Pass ID token to dryrun instances to speed things up (#6019)
* Pass ID token to dryrun instances to speed things up

* Parallelize metadata and dependency generation

* Use table schema from dryrun function
2024-08-08 12:38:43 -07:00
Sean Rose 4b8574f3bd
Fix bugs in `derived_view_schemas` SQL generator (#5592)
* Limit `derived_view_schemas` SQL generator to actual view directories.

* Fix the `derived_view_schemas` SQL generator to get the view schemas by dry-running their latest SQL and/or from their latest `schema.yaml` file.  Getting the schema from the currently deployed view wasn't appropriate because it wouldn't reflect the latest view code.

* Rename `View.view_schema` to `View.schema`.

* Change `View` so its schema dry-runs use the cloud function (CI doesn't have permission to run dry-run queries directly).

* Apply partition column filters in view dry-run queries when possible for speed/efficiency.

* Don't allow missing fields to prevent view schema enrichment.

* Only copy column descriptions during view schema enrichment.

* Only try enriching view schemas from their reference table `schema.yaml` files if those files actually exist.

* Change `main_1pct` view to select directly from the `main_remainder_1pct_v1` table, so the `derived_view_schemas` SQL generator can detect the partition column to use and successfully dry-run the view to determine its schema.

* Formalize the order `bqetl generate all` runs the SQL generators in.

* Have `bqetl generate all` run `derived_view_schemas` last, in case other SQL generators create derived views.

* Fix `Schema._traverse()` to only recurse if both fields are records.
2024-06-07 19:05:36 -07:00
Alexander f36e75ab2b
Revert "Restrict derived view schema generation to views with upstream schema…" (#4941)
This reverts commit f5ee129b63.
2024-02-01 12:08:51 -05:00
Alexander f5ee129b63
Restrict derived view schema generation to views with upstream schema files and directly copy reference schemas for simple views. (#4848)
* Refactor
* Copy reference schema directly if it's available
* Refactor default view code
2024-01-19 16:24:50 -05:00
Daniel Thorn b2a211eebc
DS-2642 - Add plan, product, and customer shipping info to stripe itemized report (#3998) 2023-10-10 10:33:31 -07:00
Anna Scholtz 20f764835f
Add --use_cloud_function option when generating SQL queries (#3565) 2023-02-03 14:10:04 -08:00
Anna Scholtz 30d7507f02
[Bug 1712332] Generate UNIONed app ping views (#3545)
* Generate UNIONed app ping views

* Address review feedback
2023-02-01 14:00:06 -08:00
Alexander Nicholson a837866bc5
Skip views to stable tables and UDFs; Dynamically extract view refs (#2663) 2022-01-17 13:38:02 -05:00
Alexander Nicholson 45655229d3
Added sql_generators script to create schema.yaml files for derived views (#2657) 2022-01-13 15:46:06 -05:00