5016a585b2
[Bug 1656910] Remove saved-session ping processing
2020-09-16 15:29:27 -04:00
0ff4ed6ce4
Use safe threaded pool for db connection
...
The previous pool is good for single threaded apps, which
we are not. This should fix the increasingly large
number of db connections we make.
2020-04-22 13:39:53 -04:00
92b0052c51
Cache aggregated datasets before insertion into database
2019-12-16 09:42:56 -08:00
82c3b1dad0
Update trim to include all batches
2019-12-05 11:59:17 -08:00
f9d3e191df
Delete tables in batches to avoid out of memory exception
2019-12-05 11:59:17 -08:00
7f2020ba56
Add trim-database command
2019-12-05 11:59:17 -08:00
019c3001cb
Add more command-line options and ability to drop tables
2019-12-05 11:59:17 -08:00
d5f37f83c5
Change num_partitions for mobile_aggregates
...
Each submission of data is < 50mb, so this should be a single file per submission date to avoid creating extra files.
2019-11-22 14:17:12 -08:00
30bfe79f5f
Add options to the cli for passing in arguments to postgres db
2019-11-18 10:18:08 -08:00
b497a47006
Bug 1596917 - Skip reading credentials file if not exists
2019-11-18 10:18:08 -08:00
52ecf7755e
Partition mobile aggregates to avoid excessive load during validation
2019-11-15 14:10:15 -08:00
35bb1c0a0d
Remove unused entrypoints
2019-11-13 14:28:45 -08:00
e6a2262df8
Use gzip instead of zlib
2019-11-13 14:03:04 -08:00
6d09dbfbbf
Add missing filter clause to avro load
2019-11-13 14:03:04 -08:00
3c415baf42
Add support for avro in mobile aggregates
2019-11-13 14:03:04 -08:00
4421895c81
Add tests for loading from avro in parquet job
2019-11-13 14:03:04 -08:00
238e206356
Use format strings in BigQuery shim
2019-11-13 14:03:04 -08:00
8e2dadc89b
Fix minor issues with avro convention and missing avro_load call
2019-11-11 12:50:44 -08:00
be711bc22c
Add option to load from avro files
2019-11-11 12:50:44 -08:00
446a168154
Remove excess list calls and use python3 features
2019-11-08 10:44:04 -05:00
21e2b2aaef
Use absolute imports for flask service
2019-11-08 10:44:04 -05:00
96f9a4de0f
Manually fix up code after 2to3
2019-11-08 10:44:04 -05:00
9a1b943497
Remove call to gevent.monkey.patch_all
...
This is unnecessary because gunicorn already calls it during application startup as per https://github.com/benoitc/gunicorn/blob/master/gunicorn/workers/ggevent.py#L47-L63 and http://www.gevent.org/api/gevent.monkey.html
2019-11-08 10:44:04 -05:00
0746b3b1c8
Run 2to3 on mozaggregator
2019-11-08 10:44:04 -05:00
b2ef0a7da7
Address review comments and fix missing module
2019-10-31 13:25:48 -07:00
9e9384a6ab
Update CLI and exercise BigQuery pathways
...
This also fetches credentials for the Postgres database using Spark for cross-platform compatibility.
2019-10-31 13:25:48 -07:00
3e5ffd6454
Add backwards compatible BigQuery shim as data source
2019-10-31 13:25:48 -07:00
3ba6545cde
Add CLI module for exposing job API
2019-09-05 13:59:53 -07:00
bc94500407
Make path configurable in parquet
2019-09-05 13:59:53 -07:00
a111bd7e5e
Add path argument to write_parquet in mobile
2019-09-05 13:59:53 -07:00
b0344836cd
Update requirements to address security concerns, remove cloudwatch logging for gcp deployment, update outdated installs
2019-05-21 14:31:34 -07:00
4848c5d6b8
Productionize mobile aggregates job (bug 1501091)
2019-01-11 15:01:45 -08:00
0aed8a5e93
Add regex matching for blacklist
...
- Prevents viewing of search scalars
2018-11-30 13:23:32 -06:00
394f5ee90b
Add search scalars to blacklist
2018-11-30 13:23:32 -06:00
e66ad26db2
Bug 1505103 - Filter out authed metrics
...
We don't want to return authed metrics if you can't select
them (because you're not authed). We now selectively show
metrics based on whether you're authed or not, and never
return blacklisted metrics.
2018-11-08 09:45:29 -06:00
36cc3f5e3e
Flake8 cleanup
2018-11-08 09:45:29 -06:00
bdfb3a3dc5
Address review feedback
2018-11-07 08:57:25 -06:00
b192763775
Remove logging; looking@you, nginx
2018-11-07 08:57:25 -06:00
fec93426de
Log using cachelogger
2018-11-07 08:57:25 -06:00
d3b7a1293c
Log to stdout/stderr
2018-11-07 08:57:25 -06:00
040ea2938c
Add logging for cache
2018-11-07 08:57:25 -06:00
aabd91e6ac
Add tests for cache clearing
2018-11-07 08:57:25 -06:00
750e78603c
Add is_authed sugar function
2018-11-07 08:57:25 -06:00
eda7caec0a
Add clear cache authed endpoint
2018-11-07 08:57:25 -06:00
1d55bd50da
Add terminal / to audience
2018-11-07 08:57:25 -06:00
46e167c3d4
Fix test with correct auth0 URL
2018-11-07 08:57:25 -06:00
18eb5fb9ed
Final changes for release aggregates
...
1. Use prod endpoint for auth0
2. Cache auth0 creds for 15 mins
3. s/401/403
2018-11-07 08:57:25 -06:00
a7e7f7ebdf
Cache authed requests separately
2018-11-07 08:57:25 -06:00
ce77b94ea4
Authorization for prerelease auth metrics
2018-11-07 08:57:25 -06:00
12c3a00914
Enable event count telemetry on release
2018-11-07 08:57:25 -06:00
Anna Scholtz
Frank Bertsch
Anthony Miyaguchi
Harold Woo
Rob Hudson