mozilla
/
python_mozaggregator
зеркало из https://github.com/mozilla/python_mozaggregator.git
1
0
Форкнуть 0
Код Задачи Пакеты Проекты Релизы Вики Активность
457 коммитов 39 Ветки 4 Теги 2 MiB
Граф коммитов

457 Коммитов

Автор SHA1 Сообщение Дата
kik-kik 6c0119bfd0 Updated config.yml 2023-01-13 18:07:33 +01:00
kik-kik 1b56535961 updated http to https for link to moztelemetry git repository 2023-01-13 16:57:39 +01:00
kik-kik 6344ecfd5e Updated config.yml 2023-01-11 22:03:08 +01:00
kik-kik 2fb35cbc0c updated python-moztelemetry requirement to be installed from git repo as the package has been removed from pypi 2023-01-11 21:55:33 +01:00
Anthony Miyaguchi 3d918a36b6 Rename branch in badge 2021-02-03 12:47:18 -08:00
Anthony Miyaguchi 670753487c Update config.yml to deploy from main 2021-02-03 12:47:18 -08:00
Anna Scholtz 8d87a7e72f Bump protobuf version 2020-09-16 15:29:27 -04:00
Anna Scholtz e16301dcc7 Fix circleci build step 2020-09-16 15:29:27 -04:00
Anna Scholtz 5016a585b2 [Bug 1656910] Remove saved-session ping processing 2020-09-16 15:29:27 -04:00
Frank Bertsch 0ff4ed6ce4 Use safe threaded pool for db connection 
The previous pool is good for single threaded apps, which
we are not. This should fix the increasingly large
number of db connections we make.
 2020-04-22 13:39:53 -04:00
Frank Bertsch 041fb3f5af Expose postgres ports 
Remove linking, which was deprecated.
This fixes the issue where the db does not start.
 2020-04-22 13:39:53 -04:00
Anthony Miyaguchi 9bc450318e Add script for validating a copy of the database 2020-03-05 11:11:42 -08:00
dependabot[bot] 207068afed Bump werkzeug from 0.14.1 to 0.15.3 in /requirements 
Bumps [werkzeug](https://github.com/pallets/werkzeug) from 0.14.1 to 0.15.3.
- [Release notes](https://github.com/pallets/werkzeug/releases)
- [Changelog](https://github.com/pallets/werkzeug/blob/master/CHANGES.rst)
- [Commits](https://github.com/pallets/werkzeug/compare/0.14.1...0.15.3)

Signed-off-by: dependabot[bot] <support@github.com>
 2020-01-28 08:09:24 -08:00
dependabot[bot] 6d60b615a8 Bump flask from 0.12.4 to 1.0 in /requirements 
Bumps [flask](https://github.com/pallets/flask) from 0.12.4 to 1.0.
- [Release notes](https://github.com/pallets/flask/releases)
- [Changelog](https://github.com/pallets/flask/blob/master/CHANGES.rst)
- [Commits](https://github.com/pallets/flask/compare/0.12.4...1.0)

Signed-off-by: dependabot[bot] <support@github.com>
 2020-01-28 07:49:20 -08:00
Anthony Miyaguchi e1a76808fb Update bin/README 2020-01-02 12:23:59 -08:00
Anthony Miyaguchi 3dfb976315 Update export-avro script to accept table name and channels 2020-01-02 12:23:59 -08:00
Anthony Miyaguchi b320351409 Add Google Cloud SDK to the docker image 2020-01-02 12:23:59 -08:00
Anthony Miyaguchi 92b0052c51 Cache aggregated datasets before insertion into database 2019-12-16 09:42:56 -08:00
Anthony Miyaguchi 82c3b1dad0 Update trim to include all batches 2019-12-05 11:59:17 -08:00
Anthony Miyaguchi f9d3e191df Delete tables in batches to avoid out of memory exception 2019-12-05 11:59:17 -08:00
Anthony Miyaguchi 7f2020ba56 Add trim-database command 2019-12-05 11:59:17 -08:00
Anthony Miyaguchi 5c0f3171b0 Add tests for job to trim database 2019-12-05 11:59:17 -08:00
Anthony Miyaguchi 019c3001cb Add more command-line options and ability to drop tables 2019-12-05 11:59:17 -08:00
Anthony Miyaguchi d5f37f83c5 Change num_partitions for mobile_aggregates 
Each submission of data is < 50mb, so this should be a single file per submission date to avoid creating extra files.
 2019-11-22 14:17:12 -08:00
Anthony Miyaguchi 30bfe79f5f Add options to the cli for passing in arguments to postgres db 2019-11-18 10:18:08 -08:00
Anthony Miyaguchi d084522ec6 Add tests for assuming credentials from environment 2019-11-18 10:18:08 -08:00
Anthony Miyaguchi b497a47006 Bug 1596917 - Skip reading credentials file if not exists 2019-11-18 10:18:08 -08:00
Anthony Miyaguchi 52ecf7755e Partition mobile aggregates to avoid excessive load during validation 2019-11-15 14:10:15 -08:00
Anthony Miyaguchi 35bb1c0a0d Remove unused entrypoints 2019-11-13 14:28:45 -08:00
Anthony Miyaguchi 8c68b48bb7 Add google-resumable-media to test dependencies 2019-11-13 14:03:04 -08:00
Anthony Miyaguchi 4e9b4d1d99 Make n1-standard-8 the default 2019-11-13 14:03:04 -08:00
Anthony Miyaguchi c89d267843 Update export script to write to a destination table 2019-11-13 14:03:04 -08:00
Anthony Miyaguchi 914bb47430 Set spark.python.profile 2019-11-13 14:03:04 -08:00
Anthony Miyaguchi e6a2262df8 Use gzip instead of zlib 2019-11-13 14:03:04 -08:00
Anthony Miyaguchi d75d296a41 Add component gateway for interacting with cluster 2019-11-13 14:03:04 -08:00
Anthony Miyaguchi 6d09dbfbbf Add missing filter clause to avro load 2019-11-13 14:03:04 -08:00
Anthony Miyaguchi d93c50bf97 Update dataproc script with cluster salt, python3 build, and max-idle 2019-11-13 14:03:04 -08:00
Anthony Miyaguchi 68ff3aae51 Add script for exporting avro files from BigQuery for testing 2019-11-13 14:03:04 -08:00
Anthony Miyaguchi 3c415baf42 Add support for avro in mobile aggregates 2019-11-13 14:03:04 -08:00
Anthony Miyaguchi eefa47e498 Fix avro fixture test and use format strings 2019-11-13 14:03:04 -08:00
Anthony Miyaguchi 4421895c81 Add tests for loading from avro in parquet job 2019-11-13 14:03:04 -08:00
Anthony Miyaguchi 238e206356 Use format strings in BigQuery shim 2019-11-13 14:03:04 -08:00
Anthony Miyaguchi ecd45950ce Update dataproc script to use python3 2019-11-11 12:50:44 -08:00
Anthony Miyaguchi 13843c68e7 Add avro loading test to test_db 2019-11-11 12:50:44 -08:00
Anthony Miyaguchi 8e2dadc89b Fix minor issues with avro convention and missing avro_load call 2019-11-11 12:50:44 -08:00
Anthony Miyaguchi b2b512c3b4 Update fixture to write avro partitioned by submission date 2019-11-11 12:50:44 -08:00
Anthony Miyaguchi be711bc22c Add option to load from avro files 2019-11-11 12:50:44 -08:00
Anthony Miyaguchi 446a168154 Remove excess list calls and use python3 features 2019-11-08 10:44:04 -05:00
Anthony Miyaguchi dd25ecafec Remove unnecessary dependencies 2019-11-08 10:44:04 -05:00
Anthony Miyaguchi 21e2b2aaef Use absolute imports for flask service 2019-11-08 10:44:04 -05:00