6a27f9dad2
chore(ci): Update validate-dags Python to 3.10.12 ( #3976 )
2023-06-22 13:48:49 +00:00
3f787df986
Fix destination paths when copying stage files in CI ( #3895 )
2023-06-02 19:06:01 +00:00
071c53e4cb
DENG-803/805: Create & Validate backfill cli commands ( #3760 )
...
* Added backfill create and validate cli ommand
---------
Co-authored-by: Alexander <anicholson@mozilla.com>
Co-authored-by: kik-kik <42538694+kik-kik@users.noreply.github.com>
2023-06-01 10:06:09 -07:00
cbe42ab9a9
Deng 850 firefox android clients reported ping ( #3789 )
...
* DENG-850 Retrieve FALSE instead of NULL in the in metadata when there isn't first_session or metrics ping.
* DENG-850 Unitest for no first session ping.
* DENG-850 syntax fix
* DENG-850 Tests for first session ping, and no baseline ping.
* DENG-850 Tests suite.
* DENG-850 YAML fixes.
* DENG-850 Adjustment to the case of reported first_session and metrics ping. The unitests are adjusted to get the value for reported pings.
* DENG-850 Add sample id to test.
---------
Co-authored-by: Lucia Vargas <lvargas@mozilla.com>
2023-05-26 12:50:08 +02:00
0b40246e92
generate sql and dags for main in a separate CI task for faster results ( #3828 )
2023-05-23 10:14:12 -07:00
a0d810275b
Remove java dependency in favor of sqlglot ( #3755 )
2023-05-17 14:56:42 -07:00
70adb102f4
Skip deploy jobs on non-main branches ( #3777 )
...
* Skip deploy jobs
* Remove tag conditions
2023-05-15 09:02:08 -04:00
dbacc6e4b3
Build Docker image in CI. ( #3759 )
...
To catch Dockerfile issues prior to merging.
2023-05-04 08:31:37 -07:00
381f95cc15
Add support for --sql-dir option ( #3708 )
2023-04-06 15:23:37 -07:00
3d11766974
Jinja related improvements ( #3693 )
2023-03-30 13:29:29 -07:00
08b45a40fe
Jinja queries support ( #3691 )
...
* Support Jinja templating in query files
* Formatting for Jinja
* ./bqetl query render command
* Fix running templates
2023-03-30 11:00:12 -07:00
51def19185
Bug 1825545 - Revert "Support Jinja templating in query files ( #3685 )" ( #3689 )
...
Bug 1825545 - This reverts commit a1c51124ec
2023-03-30 10:47:17 -04:00
a1c51124ec
Support Jinja templating in query files ( #3685 )
...
* Support Jinja templating in query files
* Formatting for Jinja
* ./bqetl query render command
2023-03-29 10:38:08 -07:00
9ef6ba1557
issues(3416): added missing metadata.yaml files inside moz-fx-data-shared-prod ( #3640 )
...
* added missing metadata.yaml files inside telemetry_derived
* added a comment where we should include a check to see if all queries have a metadata.yaml
* added the remained of missing metadata.yaml files under the moz-fx-data-shared-prod project
* fix metadata files for folders without version
* changes made as requested by ascholtz in PR#3640
---------
Co-authored-by: Leli Schiestl <lschiestl@mozilla.com>
2023-03-13 21:53:05 +00:00
60c85e7c54
Revert CI changes for private UDFs and add stub documentation - DENG-735 ( #3652 )
...
* Revert "CI fixes for supporting private UDFs in bigquery-etl - DENG-735 (#3631 )"
This reverts commit edcfe758f7
2023-03-10 11:41:46 -05:00
edcfe758f7
CI fixes for supporting private UDFs in bigquery-etl - DENG-735 ( #3631 )
...
* Minimize stub normalize_search_engine UDF and usage in search_clients_last_seen tests
* Move sql tests downstream of private-generate-sql and copy UDFs into sql-dir for tests
2023-03-09 16:54:42 -05:00
12336b8286
Run a nighly CI workflow after probe-scraper ( #3641 )
...
Probe-scraper takes at most 4 hours to run.
If those schemas get deployed, then this build will cache
the latest stable schemas and make them available in the
container.
2023-03-07 13:02:50 -05:00
e890be4c10
chore(ci): use unit tests from telemetry-airflow ( #3600 )
...
* use unit tests from telemetry-airflow
2023-02-22 13:24:56 -05:00
3362ea470a
Deploy changes to stage env ( #3587 )
...
* Deploy changes to stage env
* Test deploy
* Remove tests queries
* Add CLI option to remove updated artifacts
2023-02-22 09:49:01 -08:00
23b9c31e58
Update SSH key for pushing generated SQL ( #3595 )
2023-02-14 15:20:05 -08:00
a064fe246a
Update SSH doc deploy key ( #3594 )
2023-02-14 17:52:19 -05:00
ac053c326a
Update dependencies missed by dependabot ( #3566 )
2023-02-06 12:14:32 -08:00
2e994cc19a
Re-enable gh-pages for docs ( #3556 )
2023-02-01 10:19:33 -08:00
c68a89b871
Disable gh-pages ( #3531 )
2023-01-25 17:14:59 -08:00
fffa763f88
Fix compatibility with dependabot ( #3512 )
...
by updating pip-tools and indicating no packages are unsafe
and update google-cloud-bigquery to fix dependabot version conflict error
2023-01-17 14:14:54 -08:00
653d5bcb5e
Fix copying DAGs in CI ( #3511 )
2023-01-17 11:16:02 -08:00
031320dfcf
Move check for dags being up to date after DAGs have been generated ( #3508 )
2023-01-17 08:49:29 -08:00
5e42daae7f
Update bigquery-etl deploy key ( #3492 )
2023-01-09 22:54:43 +01:00
ef4bd86d9b
Update private-bigquery-etl deploy key ( #3491 )
2023-01-09 22:12:38 +01:00
a2ec2afeec
enforce isort in CI ( #3450 )
2022-12-12 15:11:38 -08:00
797ac94268
Update CI to python 3.10 ( #3248 )
2022-10-05 11:32:53 -07:00
81fbdf6c01
Move doc validation and generation commands to CLI
2022-08-25 08:51:41 -07:00
9a5a98779a
Remove scripts that have been replaced by CLI
2022-07-19 08:08:49 -07:00
f5b1b0d61c
Make link to manual trigger easier to copy/paste ( #3015 )
...
The link is currently followed by a '.' and it is easy to miss the next line of instruction. Indentation should make it easier to see these are grouped together.
2022-06-14 15:31:45 +00:00
5ea2f1dea2
Remove redundant GOOGLE_APPLICATION_CREDENTIALS in integration ( #2973 )
2022-05-16 19:53:50 +00:00
3fb6ebc160
Specify pyjnius in `requirements.in` ( #2924 )
2022-04-29 12:23:47 -07:00
0235d6087c
Reapply "Update to zetasql 2022.02.1 ( #2862 )" ( #2881 )
2022-04-07 14:02:20 -07:00
22835e5706
Revert "Update to zetasql 2022.02.1 ( #2862 )" ( #2879 )
...
This reverts commit 9c9761feac
2022-04-07 12:03:09 -07:00
9c9761feac
Update to zetasql 2022.02.1 ( #2862 )
2022-04-05 18:55:52 +02:00
4406437be2
Use --no-deps when installing compiled requirements files ( #2752 )
2022-02-24 21:36:47 +00:00
b5f9ea4752
Update generate_sql script
2022-02-16 08:52:51 -08:00
9582f323f5
Revert "Remove verify-dags-up-to-date from CI"
...
This reverts commit cbce03eb03
2022-02-15 11:39:27 -08:00
72d482210b
Revert "Create directory when generating DAGs"
...
This reverts commit 60888156b0
2022-02-15 11:39:27 -08:00
60888156b0
Create directory when generating DAGs
2022-02-14 08:32:04 -08:00
cbce03eb03
Remove verify-dags-up-to-date from CI
2022-02-14 08:32:04 -08:00
11139420c4
Copy generated DAGs when pushing generated SQL
2022-02-02 14:43:18 -08:00
c50e8615cf
Test DAG SQL copy
2022-02-01 11:47:26 -08:00
10ec0a4510
Copy generated DAGs to /tmp/workspace
2022-01-28 08:07:15 -08:00
cd5fefde6b
Push DAGs to generated-sql branch
2022-01-19 10:11:47 -08:00
a2728ee4ed
tweaked comment in validate-dags to mention that tags are also validated ( #2655 )
...
Co-authored-by: = <=>
2022-01-19 14:20:22 +00:00
Mikaël Ducharme
Anna Scholtz
Winnie Chan
Lucia
Daniel Thorn
Alexander
Sean Rose
kik-kik
Frank Bertsch
akkomar
Jeff Klukas