mozilla
/
bigquery-etl
зеркало из https://github.com/mozilla/bigquery-etl.git
1
0
Форкнуть 0
Код Задачи Пакеты Проекты Релизы Вики Активность
2 463 коммитов 321 ветка 1 297 Теги 223 MiB
Граф коммитов

2463 Коммитов

Автор SHA1 Сообщение Дата
Anthony Miyaguchi bd2e2730c4
Revert "Bug 1720938 - Allow ignore on unknown values during stripe import (#2197)" (#2199) 
This reverts commit 9e2470f19b.
 2021-07-19 10:06:11 -07:00
Ben Wu a441b34560
Cast subsession hours as float in search query (#2198) 2021-07-19 15:43:24 +00:00
Anna Scholtz 2f0e3672f1 Fix SQL formatting for concatenation operator 2021-07-19 08:27:23 -07:00
Ben Wu 9d063e4108
Bug 1716074 - Derive search clients daily from clients daily (#2127) 2021-07-19 14:30:35 +00:00
Anthony Miyaguchi 9e2470f19b
Bug 1720938 - Allow ignore on unknown values during stripe import (#2197) 2021-07-16 15:53:02 -07:00
Anthony Miyaguchi ec53197b71
Bug 1720767 - Adjust apple app store connect to 20:00 UTC (#2196) 2021-07-16 11:24:16 -07:00
Anthony Miyaguchi c8cae2dfbb
Bug 1720752 - Fix stripe import for custom_field array (#2193) 
* Add failing test for custom_fields in invoice object

* bug 1720752 - Fix failed stripe import due to custom_fields
 2021-07-15 13:44:21 -07:00
Jeff Klukas 42aba8436b
Small update to testing docs (#2184) 2021-07-15 19:11:15 +02:00
Frank Bertsch e6cd0227f6 Ignore null services 2021-07-15 11:48:54 -04:00
Frank Bertsch 4da220787d Add documentation about backfilling 
Co-authored-by: Jeff Klukas <jklukas@mozilla.com>
 2021-07-15 11:48:54 -04:00
Frank Bertsch 415e060277 Don't include future dates on backfill 2021-07-15 11:48:54 -04:00
Frank Bertsch 62ae592bab Make services ordering deterministic 
Co-authored-by: Jeff Klukas <jklukas@mozilla.com>
 2021-07-15 11:48:54 -04:00
Frank Bertsch 478ccc2481 Replace table if exists 2021-07-15 11:48:54 -04:00
Frank Bertsch 86d96eecea Avoid using array concat operator 2021-07-15 11:48:54 -04:00
Frank Bertsch 69f1bf5dcf Update DAG; reformat SQL 2021-07-15 11:48:54 -04:00
Frank Bertsch 1f03d4fa5f Add fxa_users_first_seen 2021-07-15 11:48:54 -04:00
dependabot[bot] 5a4cc07b12 Bump gcsfs from 2021.6.1 to 2021.7.0 
Bumps [gcsfs](https://github.com/dask/gcsfs) from 2021.6.1 to 2021.7.0.
- [Release notes](https://github.com/dask/gcsfs/releases)
- [Commits](https://github.com/dask/gcsfs/compare/2021.06.1...2021.07.0)

---
updated-dependencies:
- dependency-name: gcsfs
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
 2021-07-14 17:28:08 +00:00
Anthony Miyaguchi e2820e1255
Add filter for large scalars in aggregates (#2176) 2021-07-13 17:19:06 +00:00
Anna Scholtz b60d4f4be2 CircleCI build check for fork 2021-07-12 14:10:20 -07:00
Anna Scholtz aafa54c346 Add separate CI step for SQL and routine tests 2021-07-12 14:10:20 -07:00
Daniel Thorn 16ef74898e
Update google-api-core to 1.31.0 (#2183) 2021-07-12 18:22:08 +00:00
dependabot[bot] f123ad350d
Bump stripe from 2.58.0 to 2.59.0 (#2180) 2021-07-12 18:01:27 +00:00
Daniel Thorn d456465aa1
Update vpn available geos (#2182) 2021-07-12 17:50:38 +00:00
akkomar 374325adc7
Fix private SQL generation (#2177) 2021-07-09 14:18:17 +00:00
akkomar 9c313fe1dd
Push private generated SQL to private repository (#2174) 2021-07-09 13:57:36 +02:00
Anthony Miyaguchi 923969666a
Fix #2172 - Add canceled_for_customer_at in schema (#2175) 2021-07-08 18:21:12 +00:00
Anthony Miyaguchi 7a7cd3cae6
Bug 1719188 - Fix overflows by filtering out large histogram values (#2173) 
* Add filter on max histogram bucket value for glean glam queries

* Add update generated SQL

* Update comment
 2021-07-08 11:11:08 -07:00
Anna Scholtz fc7cd7f624 Install pyarrow for iprospect import 2021-07-07 09:43:23 -07:00
Anna Scholtz 43b2e58de6 Add date parameter for iprospect import 2021-07-06 15:49:55 -07:00
Anna Scholtz 9f4b07ff67 Add script to import iProspect data 2021-07-06 15:15:14 -07:00
dependabot[bot] 497b6758f5
Bump zipp from 3.4.1 to 3.5.0 (#2164) 2021-07-06 19:31:18 +00:00
dependabot[bot] 34a03889d3
Bump google-cloud-storage from 1.39.0 to 1.40.0 (#2161) 2021-07-06 19:18:45 +00:00
dependabot[bot] ee9528df7f
Bump importlib-metadata from 4.5.0 to 4.6.1 (#2163) 2021-07-06 19:05:24 +00:00
dependabot[bot] 45fa7d8670
Bump pandas from 1.2.4 to 1.3.0 (#2162) 2021-07-06 18:20:00 +00:00
Daniel Thorn 7045922816
Fix jvm startup race condition (#2167) 2021-07-06 18:04:04 +00:00
dependabot[bot] 72495fd98c
Bump pip-tools from 6.1.0 to 6.2.0 (#2149) 2021-07-06 17:22:52 +00:00
Anna Scholtz 92281854cd Use ExternalTaskCompletedSensor in generated Airflow DAGs 2021-07-06 08:34:26 -07:00
Anna Scholtz ed7d9dd360 [Bug 1719168] Limit max pings in clients_daily to 150000 2021-07-05 09:30:43 -07:00
Anna Scholtz fe0ab35568 Use circleci-agent step halt 2021-06-24 13:56:53 -07:00
Anna Scholtz f3203ddd8c Update docs and add pull request template 2021-06-24 13:56:53 -07:00
Anna Scholtz 3d19b616c1 Fix GH workflow 2021-06-24 09:41:44 -07:00
Jeff Klukas edf3ace296
Disambiguate ad_clicks in addons_daily query (#2156) 
* Disambiguate ad_clicks in addons_daily query

The addons_daily_v1 query failed last night with an error about `ad_clicks`
being an ambiguous field name. See
[Airflow logs](https://workflow.telemetry.mozilla.org/log?execution_date=2021-06-23T04%3A00%3A00%2B00%3A00&task_id=telemetry_derived__addons_daily__v1&dag_id=bqetl_addons).

This is due to https://github.com/mozilla/bigquery-etl/pull/2151 which added
a field called ad_clicks to clients_daily and downstream tables. That new field
is present in a join.

This PR disambiguates the reference by specifying which table to pull it from.

* Allow dry run for search queries
 2021-06-24 15:59:58 +00:00
Anna Scholtz e4ba6e1ec8
Fix cd in Github workflow (#2154) 
Co-authored-by: Jeff Klukas <jklukas@mozilla.com>
 2021-06-24 15:45:02 +00:00
Anna Scholtz 1c8180b085 Use ExternalTaskCompletedSensor for bqetl_public_data_json DAG 2021-06-24 08:34:20 -07:00
Daniel Thorn 287b41b489
Add vpn all subs field needed to identify cancel reasons (#2147) 2021-06-23 17:49:13 -04:00
Ben Wu c718833ac9
Bug 1716074 - Add search related probes to clients_daily (#2151) 2021-06-23 20:51:23 +00:00
Anna Scholtz e0f0e4a983 Fix GH action 2021-06-23 12:56:05 -07:00
Daniel Thorn 1997f14828
Add tax_behavior field to stripe price schema (#2152) 2021-06-23 10:50:35 -07:00
Anna Scholtz dfdf195823 CircleCI manual-trigger-required-for-fork with parameter 2021-06-22 15:41:42 -07:00
Anna Scholtz 3af4b612aa Add CI step for integration trigger instructions 2021-06-22 15:41:42 -07:00