0f967c4b81
Output message if no data files are found for date ( #21 )
2020-10-13 14:52:12 -04:00
5d5b16966d
Set lower chunksize for upload to gcs ( #18 )
2020-07-08 15:08:22 -04:00
3fc83e224b
Ingest streaming data from s3 ( #17 )
2020-06-30 16:17:33 -04:00
8675debdcb
Set schemas for external csv tables ( #16 )
2019-12-19 17:46:10 -05:00
ea4ed32763
Add destination dataset name to temp table name ( fixes #11 ) ( #13 )
2019-12-12 12:36:49 -05:00
4bdbd23b68
Increase max deletion size to 5000 ( fixes #8 ) ( #12 )
2019-12-11 13:43:57 -05:00
6ba1f2e249
Fix linting errors
2019-11-19 12:25:53 -05:00
19922b50b7
Insert data into BQ native tables
...
See bug 1593268.
- Add an option for removing columns before load
- Partition table based on load date
- Delete partition before loading
- Create table on first load
- Move clients to be class vars
- Update tests
2019-11-19 12:22:11 -05:00
3db14f802a
Fix tests for new param
2019-10-21 20:41:13 -04:00
5374f876aa
Avoid storage.buckets.get API call
2019-10-21 13:22:25 -07:00
c7d516a97d
Add optional version number
...
This will deal with possible schema evolution issues. We will
simply increment the version and deal with it in a view.
This also requires table_prefix and version when
calling export, both of which are given defaul values
in the CLI.
2019-10-16 13:47:45 -04:00
f57a211954
Remove output from file and table names
2019-10-16 10:44:51 -04:00
cefd996b6a
Add option for tablename prefix
2019-10-16 10:14:52 -04:00
cb9e41c1ac
Lint; add ci linting
2019-10-14 10:20:52 -04:00
88a1a4c88b
Drop table if exists before creation
2019-10-14 10:09:48 -04:00
c3c5f9d417
Delete GCS prefix before writing
2019-10-14 10:05:11 -04:00
5265071bd5
Add python export mechanism
2019-10-11 14:09:59 -04:00
3380ceca92
Remove application leftovers
2019-10-11 14:09:21 -04:00
1fa6dc869e
Add initial skeleton for job
2019-10-08 11:07:55 -04:00
Ben Wu
Frank Bertsch
Wesley Dawson