a6aa8d8db5
Schedule monitoring tasks in bqetl_monitoring
2020-11-13 10:39:34 -08:00
9923823ddd
Change GA dag to run later ( #1515 )
2020-11-04 14:47:32 -05:00
1ace0fe2b7
Ran YAMLlint on all yaml files and resolved linting issues ( fixes #1297 ) ( #1481 )
...
* "Ran YAMLlint on all yaml files"
* "Moved product info metadata table to README file"
* "Reformatted yaml lists"
* "Updated line breaks so script runs"
* "Updated line breaks so script runs"
* "Undid line breaks"
* "Created custom config file"
* "Removed base document id"
* "Undid line breaks"
* "Reformatted code"
* "Trimmed whitespace"
* "Undid line break"
* "Introduced newline"
* "Trimmed whitespace"
* "Added yamillint to config file"
* "Added yamllint to config file"
* "Moved up yamllint test"
* "Trimmed whitespace"
* "Trimmed whitespace"
* "Trimmed whitespace"
* "Trimmed whitespace"
* "Removing hyphen to fix CI error"
* "Indentation to remove CI error"
* "Included yamllint install in build run"
* "Added yamllint in requirements.txt and .in file"
* "Moved install yamllint step to its own stage"
* "Updated yamllint test"
* "Updated circleci step"
* "Reformatted code"
* "Added yamllint to circleci steps"
* "Added checkout block to yamllint step"
* "Trimmed whitespace"
* "Undid yamllint step"
* "Specified directory name for yamllint test"
* "Fixed yamlint errors"
* "Fixed yamllint errors"
* "Fixed yamllint errors"
* "Fixed yamllint errors"
* "Ignore pathway in linting"
* "Added ignore venv pathway during linting"
* "Updated ignore block"
* "Updated ignore block"
* "Removed ignore block"
* "Updated ignore block"
* "Indented base as a list"
* "Indented base item"
* Update tests/sql/moz-fx-data-shared-prod/search_derived/mobile_search_clients_last_seen_v1/test_day_bit_shifting/expect.yaml
Co-authored-by: Anthony Miyaguchi <acmiyaguchi@gmail.com>
* "Resolved linting errors"
* "Referenced tables put back on same line"
* "Fixed linting error"
* Update sql/moz-fx-data-shared-prod/account_ecosystem_derived/fxa_logging_users_daily_v1/metadata.yaml
Co-authored-by: Anthony Miyaguchi <acmiyaguchi@gmail.com>
* "Fixed linting error"
Co-authored-by: Anthony Miyaguchi <acmiyaguchi@gmail.com>
2020-10-29 17:24:55 -07:00
1529510c6e
Add derived tables for blog.m.o google analytics ( #1492 )
2020-10-28 17:40:45 -04:00
2521f926e7
Add daily schedule for geckoview versions ( #1447 )
...
* Add daily schedule for geckoview versions
* Remove unnecessary parameters
2020-10-16 14:37:30 -07:00
6ca03e713e
Export mozilla vpn waitlist from cloud SQL ( #1397 )
2020-10-09 10:18:30 -07:00
88ed89bd2c
Add stripe ETL to support Mozilla VPN dashboard ( #1349 )
2020-10-05 15:10:01 -07:00
ad896eed82
Remove anomdtct
2020-10-02 10:31:42 -07:00
d539fafb59
Bug 1635906 Add bqetl support for scripts and script for AET lookup ( #1323 )
...
* Bug 1635906 Add bqetl support for scripts and script for AET lookup
There are some code changes here for DAG generation and for testing.
* Apply suggestions from code review
Co-authored-by: Anna Scholtz <anna@scholtzan.net>
* DAG fixups
Co-authored-by: Anna Scholtz <anna@scholtzan.net>
2020-09-23 15:28:25 -04:00
818f680052
Create DAG for events rollup ( #1301 )
...
* Create DAG for events rollup
* Update sql/org_mozilla_firefox_derived/events_daily_v1/metadata.yaml
Co-authored-by: Anna Scholtz <anna@scholtzan.net>
Co-authored-by: Anna Scholtz <anna@scholtzan.net>
2020-09-10 18:50:45 -04:00
b0efb898c5
Add bqetl_fxa_events DAG ( #1163 )
...
* Add bqetl_fxa_events DAG
Replaces the fxa_events DAG in telemetry-airflow
Co-authored-by: Anna Scholtz <anna@scholtzan.net>
2020-07-14 07:58:11 -04:00
2fa01c6ddb
bqetl_asn_aggregates
2020-07-13 11:15:30 -07:00
2d8e08294b
Add query for monitoring deletion request volume ( #1106 )
2020-07-13 11:11:06 -07:00
41b05f58d3
Rescheduling
2020-07-10 13:30:24 -07:00
ee4a3ee0ce
More task re-scheduling
2020-07-10 13:30:24 -07:00
c4054ccdbf
Reschedule and set main_summary priority
2020-07-09 13:08:17 -07:00
c2d1430ede
Reschedule DAGs
2020-07-08 13:46:03 -07:00
75ad93c9cd
Calculate execution delta for tasks
2020-07-08 13:46:03 -07:00
4d85266015
Refactoring
2020-07-06 11:27:26 -07:00
c5d727c336
DAG for daily experiments clients
2020-07-06 11:27:26 -07:00
e444d450b6
Support multipart queries and main_summary
2020-07-06 11:27:26 -07:00
76bac7a98e
Create an ETL job for the Internet Outages ( #1058 )
...
* Add aggregation by country
* Copy the initial Italy focus query
This initial commit provides a baseline for the
next commits to ease review, since this initial
code was already reviewed.
* Cleanup the country list and replace FULL OUTER with LEFT joins
* Aggregate by city for cities with more than 15k inhabitants
The actual 15k limit is enforced at ingestion time.
This further limits the resulting cities to ones with at
least 1000 active daily users.
* Produce hourly aggregates
* Move the query to the `internet_outage` dataset
* Provide automatic daily scheduling through AirFlow
* Tweak the SQL addressing review comments
This additionally changes the `CAST` to
`SAFE_CAST` to account for weirdnesses in
the data.
* Add ssl_error_prop
* Add missing_dns_success
* Add missing_dns_failure
* Lower the minimum reported bucket size to 50
This allows us to match the EDA by Saptarshi and
to have a better comparable baseline.
* Document the oddities around `submission_timestamp_min`
2020-07-01 06:44:40 +02:00
49ee9b34b1
Rename bqetl_clients to bqetl_clients_daily
2020-06-24 12:18:04 -07:00
38e6acbee6
DAGs for client queries
2020-06-24 12:18:04 -07:00
48430fcfe1
Move addons queries to bqetl_addons DAG
2020-06-24 08:54:22 -07:00
45fb7d41e5
Add bqetl_search DAG
2020-06-12 09:47:51 -07:00
d72918d6e7
Create bqetl_activity_stream DAG
2020-06-12 08:32:42 -07:00
038b4b18e7
DAG for messaging system queries
2020-06-10 13:45:22 -07:00
43e806c9e7
Move smoot queries to bqetl_gud DAG
2020-06-09 09:38:11 -07:00
69d68c55c3
bqetl_mobile_search DAG
2020-06-08 13:33:19 -07:00
823542b235
bqetl_nondesktop DAG
2020-06-08 12:36:06 -07:00
494f8c1a5f
bqetl_core DAG
2020-06-08 12:05:26 -07:00
b370c4710d
DAG for vrbrowser queries
2020-06-05 10:52:12 -07:00
5b906f859c
Add version to error_aggregates query
2020-06-03 14:39:51 -07:00
5cbdfa0dcc
Add bqetl_amo_stats DAG
2020-06-03 14:28:59 -04:00
914e74c9ce
Add support for defining external task dependencies
2020-06-03 09:32:50 -04:00
3a57c323ce
Use consistent naming for DAG publishing JSON
2020-06-02 11:04:32 -07:00
45961c331b
Handle public data export in DAGs
2020-06-02 11:04:32 -07:00
ca287bdf3a
Keep error_aggregates query without version
2020-05-28 14:12:24 -07:00
e7b2b56c01
KPI dashboard generated Airflow DAG
2020-05-28 14:12:24 -07:00
d5822b952d
Generate error aggregates DAG
2020-05-28 14:12:24 -07:00
22f1c39b5b
Error aggregates as scheduled query
2020-05-28 14:12:24 -07:00
1116be16b6
Pull in telemetry-airflow
2020-05-28 14:12:24 -07:00
Anna Scholtz
Ben Wu
Rhys
Anthony Miyaguchi
Daniel Thorn
Jeff Klukas
Frank Bertsch
Alessio Placitelli
Jeff Klukas