Граф коммитов

95 Коммитов

Автор SHA1 Сообщение Дата
Frank Bertsch 0ff4ed6ce4 Use safe threaded pool for db connection
The previous pool is good for single threaded apps, which
we are not. This should fix the increasingly large
number of db connections we make.
2020-04-22 13:39:53 -04:00
Anthony Miyaguchi 446a168154 Remove excess list calls and use python3 features 2019-11-08 10:44:04 -05:00
Anthony Miyaguchi 21e2b2aaef Use absolute imports for flask service 2019-11-08 10:44:04 -05:00
Anthony Miyaguchi 96f9a4de0f Manually fix up code after 2to3 2019-11-08 10:44:04 -05:00
Anthony Miyaguchi 9a1b943497 Remove call to gevent.monkey.patch_all
This is unnecessary because gunicorn already calls it during application startup as per https://github.com/benoitc/gunicorn/blob/master/gunicorn/workers/ggevent.py#L47-L63 and http://www.gevent.org/api/gevent.monkey.html
2019-11-08 10:44:04 -05:00
Anthony Miyaguchi 0746b3b1c8 Run 2to3 on mozaggregator 2019-11-08 10:44:04 -05:00
Harold Woo b0344836cd Update requirements to address security concerns, remove cloudwatch logging for gcp deployment, update outdated installs 2019-05-21 14:31:34 -07:00
Frank Bertsch 0aed8a5e93 Add regex matching for blacklist
- Prevents viewing of search scalars
2018-11-30 13:23:32 -06:00
Frank Bertsch 394f5ee90b Add search scalars to blacklist 2018-11-30 13:23:32 -06:00
Frank Bertsch e66ad26db2 Bug 1505103 - Filter out authed metrics
We don't want to return authed metrics if you can't select
them (because you're not authed). We now selectively show
metrics based on whether you're authed or not, and never
return blacklisted metrics.
2018-11-08 09:45:29 -06:00
Frank Bertsch 36cc3f5e3e Flake8 cleanup 2018-11-08 09:45:29 -06:00
Frank Bertsch bdfb3a3dc5 Address review feedback 2018-11-07 08:57:25 -06:00
Frank Bertsch b192763775 Remove logging; looking@you, nginx 2018-11-07 08:57:25 -06:00
Frank Bertsch fec93426de Log using cachelogger 2018-11-07 08:57:25 -06:00
Frank Bertsch d3b7a1293c Log to stdout/stderr 2018-11-07 08:57:25 -06:00
Frank Bertsch 040ea2938c Add logging for cache 2018-11-07 08:57:25 -06:00
Frank Bertsch aabd91e6ac Add tests for cache clearing 2018-11-07 08:57:25 -06:00
Frank Bertsch 750e78603c Add is_authed sugar function 2018-11-07 08:57:25 -06:00
Frank Bertsch eda7caec0a Add clear cache authed endpoint 2018-11-07 08:57:25 -06:00
Frank Bertsch 1d55bd50da Add terminal / to audience 2018-11-07 08:57:25 -06:00
Frank Bertsch 46e167c3d4 Fix test with correct auth0 URL 2018-11-07 08:57:25 -06:00
Frank Bertsch 18eb5fb9ed Final changes for release aggregates
1. Use prod endpoint for auth0
2. Cache auth0 creds for 15 mins
3. s/401/403
2018-11-07 08:57:25 -06:00
Frank Bertsch a7e7f7ebdf Cache authed requests separately 2018-11-07 08:57:25 -06:00
Frank Bertsch ce77b94ea4 Authorization for prerelease auth metrics 2018-11-07 08:57:25 -06:00
Frank Bertsch 12c3a00914 Enable event count telemetry on release 2018-11-07 08:57:25 -06:00
Frank Bertsch ad47959239 Prefix constants with AUTH0 2018-11-07 08:57:25 -06:00
Frank Bertsch 2ca728a01c Remove talisman; add headers explicitly 2018-11-07 08:57:25 -06:00
Frank Bertsch 79f4b7d800 Add authorization header 2018-11-07 08:57:25 -06:00
Frank Bertsch ece0d725c7 Add Talisman for CORS headers 2018-11-07 08:57:25 -06:00
Frank Bertsch dfaf2d3b1b WIP: Authorization for release aggregates 2018-11-07 08:57:25 -06:00
Frank Bertsch 5322ba5212 Address review feedback 2018-08-24 10:42:10 -05:00
Frank Bertsch aa11f09bb6 Prevent public viewing of release data
To enable release data, there are two options:
1. Add the probes you wish to display to PUBLIC_RELEASE_METRICS
2. Set the $SHOW_RELEASE_METRICS env var to "True" (and all
   probes will be displayed, with no restrictions)
2018-08-24 10:42:10 -05:00
Frank Bertsch 72873a3999 Add 404 descriptions for easier debugging 2018-08-24 10:42:10 -05:00
Rob Hudson e7194b5cde Add `osVersion` to allowed query strings (bug 1481832) 2018-08-15 09:08:04 -07:00
Frank Bertsch f628b2adff Blacklist dynamic event summaries 2018-05-02 11:38:21 -05:00
Frank Bertsch c09a6d8b1a Bug 1451779: 404 event summaries 2018-05-02 11:38:21 -05:00
Rob Hudson fb1e249e3a Bump python_moztelemetry version and update code 2018-04-10 15:30:24 -07:00
Rob Hudson 8ace3284af Remove e10s from aggregates dimensions (bug 1441586) 2018-03-29 21:13:35 -07:00
Rob Hudson 386ca7177d Misc code cleanup 2018-03-29 21:13:35 -07:00
Rob Hudson bd1e131795 Don't use cloudwatch logs in development 2018-03-14 16:02:44 -07:00
Rob Hudson 4672fd68a2 Use nightly histogram URL as default 2018-03-14 09:32:03 -05:00
Rob Hudson e4476c92db Cleanup on aisle pep8 2018-03-14 09:32:03 -05:00
Harold Woo b14172c9a5 [Bug 1342947] - Refactor mozaggregator service to use flask dockerflow 2018-03-09 10:39:33 -08:00
Frank Bertsch 4321495521 Make missing scalars 404
Bug 1432547
2018-01-23 12:55:41 -06:00
Chris H-C ad74f877c7 Fix Use Counters at the Service Level
bug 1412382

Use counters in the database are wrong, and have been since bug 1204994 changed
_extract_main_histograms. It only counted "False" values in pings that had at
least one "True" value (because if there were only "False" values, we didn't
send a use counter in that ping).

This fixes this by subbing in the False value from
(TOP_LEVEL_)CONTENT_DOCUMENTS_DESTROYED, which has the correct number.

This results in an interesting side-effect that use counters that don't exist
will get valid responses from the service. This is because the service can't
tell the difference between a use counter that doesn't exist and one that just
didn't happen to have a single 'True' value in that row in that table.

This is beneficial for testing, so that I don't have to manipulate
histograms_template or generate_payload in dataset.py to be able to support
probes that aren't in every ping. The math for expected_count in test_db
would only get worse.

Instead I can ask for a use counter that doesn't exist and ensure that it
reports the correct number of False values and the correct count.
2018-01-17 10:07:03 -05:00
Frank Bertsch 3c286a59c1 Ignore stream describe throttling 2017-05-22 11:26:16 -05:00
Frank Bertsch 0859301595 Ignore cloudwatch log upload failures
Also fixes a bug where the response from cloudwatch logs is
never set.
2017-05-15 18:35:47 +01:00
Frank Bertsch cdf5e74001 Add read replica for service 2017-03-14 07:49:05 +00:00
Frank Bertsch 817c6b8f1f Use moztelemetry Scalar implementation 2017-03-08 11:20:04 -05:00
Frank Bertsch 88b781ab5d Add Cache Control and ETag headers
This change implements both cache control and ETag headers.

For Cache-Control:
For all requests but submission-date aggregates, the max-age
is set until the data is kicked from the local cache
(we know the response won't change until then).

For submission-date aggregates, max-age is always set to 24h.

ETags:
Etags are not set on any requests but submission-date aggregates.
The ETags are the same for all values, since submission-date
aggregates will never change, unless we do a backfill.

Thus, the single ETag value can be updated, invalidating all
previous ETags. This should only be done after a backfill.
2017-03-07 11:59:06 -05:00