Граф коммитов

733 Коммитов

Автор SHA1 Сообщение Дата
Adrian Utrilla c8e1a28208
Use utf8mb4 for connections to MySQL in dev 2019-01-18 17:29:32 +01:00
Mathieu Pillard 70760159c3 Replace urlparse imports by six.moves.urllib_parse for Python 3 compatibility
This keeps backwards-compatibility to Python 2 but moves us closer to running
with Python 3.
2019-01-07 17:43:17 +01:00
Christopher Grebs 2c82ea7728
Enable memcached by default for tests. Upgrade python-memcached
* Adds a test to test unicode support for latest python-memcached
  versions
* Configures caches key prefix so that tests don't conflict with each other
* Remove a few cache.clear calls with `cache.delete` calls that avoids
deleting all memcached entries while other tests are running
* Remove zadmin.memcached as it's crazy dangerous, such power should
only be wielded by ops-folks who wake up at night when pressed
* Fixes waffle helpers to call .flush() to flush appropriate caches

This also fixes how parameters are forwarded to memcached and fix KEY_PREFIX initialization

We are now making sure parameters from `settings.CACHES` are properly forwarded
to the actual cache backend (now memcached). These errors were not seen
since LocMemCache didn't require any parameters.

The `KEY_PREFIX` setting is set to a uuid instead of the xdist-slaveid to
ensure test functions are properly isolated from each other.

`cache.clear` is not called anywhere unless absolutely necessary to reduce
the side-effects of our test-setup.

Fixes #8602
2018-11-21 18:26:30 +01:00
William Durand e6ed0f3371
Use the right domain in each env (#9727) 2018-10-17 17:52:21 +02:00
Christopher Grebs 5655295e05
Run validator via subprocess, huge cleanup along the way (#9682)
This ports all amo-validator related code to run via subprocess. This
makes us less blocked by it not running on Python 3, there may still be
some installation / execution related bits to actually run it via python
2 instead of python 3 but that's something for later.

* Cleans up how and where we're setting _BIN settings for all the
binaries we're calling
* Backports `unicodehelper` from validator
* Removes `requeue_uploads` since it's been broken and unused for at
least 3-4 years
* Remove Standalone Add-on Compatibility Checker (/developers/addon/check-compatibility/)
* Remove compat-check related bits from standalone add-on upload
* Remove more bulk compat report stuff that isn't used anymore
* Removed zadmin application_versions.json view, unused

Fixes #9670
"Fixes" #8857
2018-10-16 08:30:16 +02:00
Mathieu Pillard d7876929b3 Remove filesystem cache, use default (memcached) backend for everything
Only the language tools API used the filesystem cache, and that was
because it had a relatively low usage compared to cache-machine, which
meant everything it cached was quickly evicted. Now that cache-machine
is gone, let's use memcached again for this.
2018-07-27 11:35:42 +02:00
Christopher Grebs a2ddf0d5f4
Avoid logging through mozlog in tests experiment (#8528)
Closes #8451 (Doesn't necessary fix it, we don't want a unified logging-level but this PR makes logging much clearer.)

* Don't use verbose mode for tox
* Refactor logging settings
* Fix services logging, correct dev and stage settings, update docs
* Update logging
* Be vigilent about disabling all logging
* Remove MOZLOG_NAME from stage settings
2018-06-18 06:10:26 -07:00
Christopher Grebs 46087ea63e
Completely remove traces of caching.backends settings.
Most of this work landed already in
6a8fc341da and
91605a4c1f so this is just a follow-up to
remove all other traces.

The removed comment is no longer true, Django received all the necessary
fixes for LocMemCache. On the other hand, LoMemCache is no longer used
for a regular local setup because we are using memcached and try to be
as close to production as possible so this won't change anytime
soon.
2018-06-16 22:56:10 -07:00
Christopher Grebs 7949c933be
Report if there are any errors in local_settings.
This will issue a `UserWarning` in case some kind of exception is being
raised inside local_settings.py
2018-06-03 13:37:40 +02:00
Christopher Grebs c1112d89a1
WIP: Cleanup locustio based performance / smoke tests, add more entities (#8358)
* Port mozilla/amo-loadtest to FXA authentication and WebExtensions.

This is the first part of many to improve our load-tests and allow them
to be run regularly as part of a regression test suite.

This commit primarily ports the existing tests over to our repository,
updates them to use FxA authentication and uses a WebExtension for the
upload test.

This only implements the baseline for more work to come in #7744 but
it's important to have authentication and the basic infrastructure
working correctly.

It's also still using the legacy add-on detail and listing pages,
that'll obviously change - or be only true for SeaMonkey and Thunderbird
related tests since we still have to support them (very low priority though)

It also adds a first step of a summary report that links to new-relic.
That'll need a bit more tooling and testing but it worked quite well in
first tests.

Refs #7744

Next steps:
 * Implement most of the read-only tests from #7744
 * Check what needs auth in #7744, implement it properly
 * Implement database and cache query logging
 * Implement a unified merged test-report that uses the database and
 cache query logging and merges it with locust logs (using our new
 unique request id)

* Fix makefile, show dummy-usage for now.

* Delete generate-summary script for now

* Pin and cleanup dependencies

* Reverse all urls

* Use passed in account for login

* split things up

* Use gevent for waiting, some cleanups, add first version of Dockerfile, running script, disable developer stuff for now

* Add browsing collections and browsing categories

* Add loads more variation, add support for thunderbird, seamonkey, multiple languages

* Decrease size of docker images for local testing

* Get installation and docker image mostly to work, docs, cleanups

* Allow fxa environment variables be overwritten

* Hit legacy site of every variation

* More explicit theme testing

* Add browsing for app-versions and various rss feeds

* More variation in rss feeds, add featured and search tools

* Add a few more response.success() calls

* Test user profile pages, warn for empty collection pages

* More docs

* Add review browsing
2018-05-30 16:46:16 +02:00
Andrew Williamson 561f57138c
match both v3 and v4 when we're testing for api in middleware, etc (#8291) 2018-05-29 09:41:07 +01:00
William Durand cd989ed6f6
Redirect new users to edit profile in -dev (#8297) 2018-05-18 16:35:15 +02:00
Andrew Williamson cae6c3ad2f
drop mozilla-logger and all logging1.0 support (#8242)
* drop mozilla-logger and all logging1.0 support

* drop (hopefully) unused SYSLOG_CSP setting

* drop 'error' formatter; refactor 'debug' formatter format a little
2018-05-11 18:37:42 +01:00
Mathieu Pillard e01868ca5e Remove references to obsolete addon collector (retired in 2012) 2018-05-09 15:56:10 +02:00
Mathieu Pillard c3e6d3cc1f Add a setting controlling whether cache-machine is enabled or not
- Also, use that setting to disable cache-machine by default in tests,
  locally and on dev.
- Tweak translations tests to not depend on cache-machine at all
  (remove dependency on ModelBase, depend on django models only
   + the translation transformer)
2018-04-25 15:03:22 +02:00
Andrew Williamson 55bbcb1556
Simplify settings files (#8001) 2018-04-09 22:48:02 +08:00
Andrew Williamson 3faf451dca
remove Stylus (#7820) 2018-03-19 22:19:31 +08:00
Mathieu Pillard c125bf9354 Implement appversion compatibility filtering in language tools API
- Allow filtering by type to only show language packs
- Expose a new current_compatible_version if appversion is passed.
- This property will fetch the latest publicly available version
  compatible with the appversion passed.
- Replace caching with a filesystem-based implementation to work
  around cache eviction issues.
2018-03-05 15:44:46 +01:00
Andrew Williamson a00451308d
Update fxa config with new client for new redirect url 2018-02-08 16:40:45 +08:00
Christopher Grebs 17bb6968e5
First part moving to olympia.test 2018-01-26 07:11:48 +01:00
Christopher Grebs 176383b168 Upgrade to Celery 4 (#6667)
* Update celery, config and dependencies to 4.x

* Remove usage of TaskSet, replace with group(), update `ampq`
* Enable redis on travis for better celery 4.x testing
* Fix lib/celery_tests
* Correct test testing email sending retrying on errors
* Fix celery subtask mocking errors.

* Fix celery broker url env variable

* Integrate watchdog for celery autoreload

* Ensure dependencies are updated for worker and web container.

* Restart all services after the dependency update

* Remove explicit celery loader environment variable.

* Restart all services in web and worker after running 'update_docker'

* Increase sleep time a bit
2017-10-16 11:16:41 +02:00
Mathieu Pillard 43d88b4f55 Revert "Upgrade to Celery 4 (#6625)"
This reverts commit 541688d5f2.

This caused some breakage on dev. We suspect it's because newrelic
hooks into the celery task launching code but does not support
celery 4.x yet.
2017-10-13 00:04:27 +02:00
Christopher Grebs 541688d5f2 Upgrade to Celery 4 (#6625)
* Update celery config to seamlessly upgrade to 4.x

* Update celery initialization for new configs and 4.x

* Update dependencies to celery 4.x

* Fix flake8

* Fix typo

* Remove usage of TaskSet, replace with group(), update `ampq`

* Enable redis on travis for better celery 4.x testing

* Fix more test setting names

* Fix lib/celery_tests

* Correct test testing email sending retrying on errors

* Fix celery subtask mocking errors.

* Fix new line

* Fix celery broker url env variable

* Integrate watchdog for celery autoreload

* Try something weird... ui-tests

* do we need extra hosts?

* more sleep?

* Try logging, try docker run to avoid port collisions

* More debugging

* Cleanup again

* Fix logs?

* Ensure dependencies are updated for worker and web container.

* Remove not functioning log saving for onw

* Restart all services after the dependency update
2017-10-12 21:05:21 +02:00
Christopher Grebs b90bf13922 Merge pull request #5791 from mozilla/1573-implement-django-jinja
Loads of clean-ups along the way.

Fixes #1573

* Moves all helpers modules to templatetags.jinja_helpers - there are a few things in there that aren't jinja related so there'll be a follow-up to split that later
* Update Jinja2 to 2.9.6
* Add django-jinja 2.3.1
* move some jingo-minify helpers to olympia.lib to avoid any jingo imports
* unified |f and |fe helpers into one |format_html helper that always escapes everything unless the input is already |safe

* Move helpers, add requirements, fix imports.
* Give backend a proper name
* Port f and fe filters from jingo
* Make start porting reviews jinja helpers module, make first test pass
* Get a handful more tests working, get rid of get_env, more and more jingo calls.
* Disable extension matching, render .txt templates too, fixes more stats tests
* Fix csv_header template to produce some reasonable and csv parsable
* Set trim_blocks to be compatible with jingo
* Unify |f and |fe into one |format that simplifies escaping rules. Update tests.

I think having one unified |format filter is beneficial, especially
given that our current escaping rules were kinda complicated and allowed
for seriously dangerous combinations even with autoescape activated.

Now everything passed as an argument to |format is explicitly escaped
unless marked as |safe, there's no way around that anymore.

* Naive first version using |format instead of |f and |fe, might need more review.
* Fix format for unicode strings
* Rename |format to |format_html
* Move nl2br_xhtml to amo.jinja_helpers, use it in all our templates.
* Move datetime filter to our code temporarily, fix mocks, fix recursive error
* Move csrf call to django_jinja csrf_token tag, fix addons templatetags
* Fix |class_selected usage, fix render_with to get registered
* Remove hack from zadmin, fix zadmin templatetags
* Fix user delete photo test, don't rely on proper context updates that happen in order.
* Fix more nl2br tests
* Fix detail page escape, django escapes differently than jinja
* Remove context from edit_review_reply_form
* Fix date created format filter usage
* Remove templates setting user, fix can_review
* Fix collect_add_widget to not inherit the global context
* Remove happy unicode client
* Cleanup can_review, add tests
* Fix tests, django use different html character for double quote escaping
* Switch to firefox latest to check ui-tests
* Cleanup and fix do_truncate
* Remove jingo, copy jingo-minify helpers that depend on jingo to our lib for now
* Rename nl2br_xhtml -> nl2br
* Jinja 2.9 compat for reviews truncate filter
* Don't use TEMPLATE_DEBUG
* Set default value for q, happens on index page.
* Fix jinja2 formatting with newstyle gettext
2017-06-30 17:57:31 -07:00
Christopher Grebs 130436572b Revert "Refactor debug locales to match addons-frontend, prepares for rtl support (#5402)"
This reverts commit 0ffc528ae5.
2017-05-16 22:58:56 +07:00
Christopher Grebs 0ffc528ae5 Refactor debug locales to match addons-frontend, prepares for rtl support (#5402)
* Update compile-mo.sh to understand dbr and dbl locales

* Cleanup omg_new_locale, add support for dbr and dbl debug locales.

* Remove dbg locale

* Add dbr and dbl debug locales

* Add support for dbr, dbl, fix setting for rtl to be LANGUAGES_BIDI as this is what django uses these days

* Remove temporary comment

* Improve debug message

* Disable clean checking for now

* Fix dbg and dbl extracted locales

* Fix dbl generation

* Fix tests

* Revert unnecessary change to zadmin.forms

* Remove cleanness check

* Remove dbr for now, it's horribly broken.

* Various cleanups according to comments
2017-05-16 22:01:46 +07:00
Mathieu Pillard a06590e4ba Log to console using mozlog json format in dev/stage/prod 2017-04-06 13:10:38 +02:00
Andrew Williamson 5e25fb7f3b add all celery queues used in CELERY_ROUTES to CELERY_QUEUES 2017-03-22 11:17:44 +01:00
Andrew Williamson eb8887fba4 Revert "define CELERY_QUEUES (#4946)" (#4995)
This reverts commit 2ac0ba9827.
2017-03-22 11:01:39 +01:00
Andrew Williamson 2ac0ba9827 define CELERY_QUEUES (#4946)
* change CELERY_DEFAULT_QUEUE to celery
* Remove 'update_webext_descriptions_all' celery route to test default routing
2017-03-22 11:00:32 +01:00
Mark Striemer 1134acc4c9 Redirect to next_path on register from mobile pages (fixes #4742) (#4747) 2017-02-23 12:19:27 -06:00
Mark Striemer 49601660f5 Support amo frontend auth more closely to production (#4681) 2017-02-17 10:08:30 -06:00
Christopher Grebs f42f51f8b7 fix(package): update clean-css to version 4.0.2 (#4504)
* fix(package): update clean-css to version 4.0.2

With v4.0 clean-css has split it's command line tools into a separate
package.

Closes #4475

* fix paths
2017-01-30 14:18:06 +01:00
Andy McKay 8cda544825 github validate hook again 2016-12-23 11:43:57 -08:00
Christopher Grebs 717ca578ce Add configured domain to ALLOWED_HOSTS. Fixes circleci tests and our runserver. (#3918)
* Add configured domain to ALLOWED_HOSTS. Fixes circleci tests and our runserver.
* Correctly map uitests to olympia.dev
2016-11-02 16:56:10 +01:00
Andrew Williamson be18b6cbdf Make all responses to email API return a validation response. (#3650) 2016-10-04 11:53:58 +01:00
Mark Striemer b6000cfcb0 Define FxA config names in settings_base.py (#3574) 2016-09-21 10:56:39 -05:00
Mark Striemer 1ee3c80093 Support development FxA config on dev (#3540) 2016-09-20 17:08:09 -05:00
Mark Striemer 2b5d8a09fe Update incorrect FXA_CONFIG redirect_urls 2016-09-14 17:49:33 -05:00
Mark Striemer c939ccbc67 Support multiple FxA configurations (fixes #3496)
Fixes #3505
2016-09-14 17:49:25 -05:00
Andrew Williamson a0facd5236 Add API endpoint to process POST email content and add to activity log;
emails received send notification emails;
review emails and notification emails can be replied to.
2016-09-13 18:29:43 +01:00
Stuart Colville 3c3c6470f7 Remove 'self' and https://addons.mozilla.org from script-src 2016-09-09 15:01:25 +01:00
Mathieu Pillard 2e85926e59 Set SERVICES_DOMAIN and SERVICES_URL correctly for local development 2016-07-20 14:05:22 +02:00
Mark Striemer b8b7475ba7 Use redirect_url not redirect_uri in internal auth (#2511) 2016-04-27 09:54:15 -05:00
Mark Striemer c64fba4c48 Internal login JSON API (fixes #2425) 2016-04-21 15:31:31 -05:00
Mark Striemer 0cffe96db7 Internal login start endpoint (fixes #2424) 2016-04-20 13:23:28 -05:00
Mark Striemer 0d1edc19f2 Allow setting the recursion limit with RECURSION_LIMIT (fixes #1911) 2016-03-15 17:31:26 +01:00
Christopher Grebs 74f14b492a Remove VALIDATE_ADDONS setting.
There's no need for it anymore.
2016-03-14 17:20:16 +01:00
Christopher Grebs 81845f9f84 First steps towards Django 1.7 support.
* Upgrade to Django 1.7.10
* Upgrade to Jingo 0.9
* Make sure the app minimalistically runs with Django 1.7
* Upgrades django-waffle and django-cache-machine
* Upgrades to a forked hack of django-piston
* Fixes systems checks
* update redisutils config
* Get rid of Jingo monkey patch.
* Update to proper SafeData support (which now supports __html__)
2016-02-24 17:11:46 +01:00
Christopher Grebs 0cbb294201 Rename addons-validator to addons-linter and update to latest release.
Fixes #1742
2016-02-24 10:53:26 +01:00