a9452d14ed
Rename default branch to main ( #23 )
2021-02-04 15:08:09 -05:00
0f967c4b81
Output message if no data files are found for date ( #21 )
2020-10-13 14:52:12 -04:00
3476d8a0e0
Add script for getting messages ( #19 )
2020-08-20 18:01:10 -04:00
5d5b16966d
Set lower chunksize for upload to gcs ( #18 )
2020-07-08 15:08:22 -04:00
3fc83e224b
Ingest streaming data from s3 ( #17 )
2020-06-30 16:17:33 -04:00
8675debdcb
Set schemas for external csv tables ( #16 )
2019-12-19 17:46:10 -05:00
9d02d2236a
Allow quoted newlines un csv load ( fixes #10 ) ( #14 )
...
* Allow jagged rows in load from CSV job
* Allow quoted newlines in csv load
* oop
2019-12-18 13:44:03 -05:00
ea4ed32763
Add destination dataset name to temp table name ( fixes #11 ) ( #13 )
2019-12-12 12:36:49 -05:00
4bdbd23b68
Increase max deletion size to 5000 ( fixes #8 ) ( #12 )
2019-12-11 13:43:57 -05:00
862ef4adbc
Merge pull request #7 from fbertsch/write_to_bq
...
Write to BQ Native Tables
2019-11-20 09:48:00 -05:00
6ba1f2e249
Fix linting errors
2019-11-19 12:25:53 -05:00
19922b50b7
Insert data into BQ native tables
...
See bug 1593268.
- Add an option for removing columns before load
- Partition table based on load date
- Delete partition before loading
- Create table on first load
- Move clients to be class vars
- Update tests
2019-11-19 12:22:11 -05:00
0eaaac83fb
Merge pull request #6 from mozilla/bq_project
...
Add required project id param
2019-10-22 09:36:05 -04:00
4f740483a6
Run lint
2019-10-21 21:59:01 -04:00
3db14f802a
Fix tests for new param
2019-10-21 20:41:13 -04:00
123b4d6a38
Add required project id param
2019-10-21 18:26:00 -04:00
b62afe5cda
Merge pull request #5 from mozilla/write_only
...
Avoid storage.buckets.get API call
2019-10-21 16:50:47 -04:00
5374f876aa
Avoid storage.buckets.get API call
2019-10-21 13:22:25 -07:00
48a9e2a808
Merge pull request #3 from mozilla/table_prefix
...
Table prefix
2019-10-18 06:43:11 -04:00
f26a10f45f
Merge branch 'master' into table_prefix
2019-10-18 06:37:47 -04:00
aaf7c11e2a
Fix ordering of args in examples
2019-10-18 06:35:29 -04:00
ee6f2c90c3
Merge pull request #4 from haroldwoo/leanplum-gcr
...
Modifying CircleCI to push to GCR rather than dockerhub
2019-10-17 20:36:32 -04:00
3d5fb5534d
Modifying CircleCI to push to GCR rather than dockerhub
2019-10-17 17:27:30 -07:00
c7d516a97d
Add optional version number
...
This will deal with possible schema evolution issues. We will
simply increment the version and deal with it in a view.
This also requires table_prefix and version when
calling export, both of which are given defaul values
in the CLI.
2019-10-16 13:47:45 -04:00
f57a211954
Remove output from file and table names
2019-10-16 10:44:51 -04:00
7a2b5b0ab0
Update README commands
2019-10-16 10:16:18 -04:00
cefd996b6a
Add option for tablename prefix
2019-10-16 10:14:52 -04:00
e6ed60565a
Merge pull request #2 from mozilla/overwrite
...
Overwrite existing GCS prefix and tables; add linting
2019-10-14 14:09:38 -04:00
029cbcda86
Add flake8 config file
2019-10-14 12:36:44 -04:00
890461c013
Add flake8 to image
2019-10-14 10:43:59 -04:00
7ddc196712
Fix status badge
2019-10-14 10:24:36 -04:00
cb9e41c1ac
Lint; add ci linting
2019-10-14 10:20:52 -04:00
88a1a4c88b
Drop table if exists before creation
2019-10-14 10:09:48 -04:00
c3c5f9d417
Delete GCS prefix before writing
2019-10-14 10:05:11 -04:00
322be966c8
Merge pull request #1 from mozilla/add_leanplum
...
Export leanplum data to GCS
2019-10-14 09:15:47 -04:00
f5379d1eb1
Add circleci integration
2019-10-11 16:16:58 -04:00
f95edc5b54
Update requirements
2019-10-11 14:13:44 -04:00
b33c574011
Make locally running app work
2019-10-11 14:13:36 -04:00
5265071bd5
Add python export mechanism
2019-10-11 14:09:59 -04:00
3380ceca92
Remove application leftovers
2019-10-11 14:09:21 -04:00
1fa6dc869e
Add initial skeleton for job
2019-10-08 11:07:55 -04:00
b6caba69dc
first commit
2019-10-04 11:46:00 -04:00
Ben Wu
Frank Bertsch
Wesley Dawson
Harold Woo