addons-server/settings_test.py

113 строки
3.6 KiB
Python
Исходник Постоянная ссылка Обычный вид История

# -*- coding: utf-8 -*-
# ruff: noqa: F405
2014-12-19 17:58:50 +03:00
from settings import * # noqa
Upgrade Celery to 4.3 (#12073) Upgrade Celery to 4.3.0 This PR upgrades to Celery 4.3, introduces a small hack because of a regression in celery. This PR does not introduce any helpers to make use of celery workers inside the tests, we delayed that to a later point to unblock the upgrade. This PR does fix some inconsistencies with elasticsearch mocks along the way as they were making some tests fail (see comment logs), the mocks are now applied and rolled back much less than before which makes them easier to reason about and to debug. Notes: * acks_late now doesn't acknowledge in case of failures or timeouts (which is the case for index_addons). We could use task_acks_on_failure_or_timeout but I figured the new behavior should be fine. * any CELERY_ALWAYS_EAGER setting in local_settings.py must be replaced by CELERY_TASK_ALWAYS_EAGER commit log: * Implement proper wait_for_tasks helper, make TestTaskQueued work with celery worker * Revert changes removing threading code from es reindex tests * Fix es reindex test, correct always eager usage. * Small cleanups, fix docs, don't reload session worker. * Ensure docker-compose celery result doesn't use the same database as any potential tests * Monkeypatch for celery regression * Actually use our own app where all tasks are already bound to. This also enables logging * Experiment: Group celery worker tests together with elasticsearch and dont run them multitasked * Remove envlist as it doesn't add anything to our specific setup * Fix es-and-celery marker selection * Remove completely redundant tests * Fix hanging tasks, let's see what travis thinks... I'll follow up with cleanups and much more docs once tests are actually succeeding * Define celery.ping in amo.celery, use redis result backend * Cleanup celery worker implementation a bit. Mimic our usage of elasticsearch specific tests, this avoids various necessary workarounds and environment cleanups because django handles things like settings properly in the first place. * Untangle es mocks a bit, raise error in case of not properly stopped patches. * Re-implement state test by using celery manager to wait for results * Explicitly re-reinitialize the workers event loop and dont reuse anything existing * Remove celery worker thread specific code. * Reset default celery result backend setting
2019-09-18 18:36:01 +03:00
# Make sure the apps needed to test translations and core are present.
2018-04-09 17:48:02 +03:00
INSTALLED_APPS += (
'olympia.translations.tests.testapp',
'olympia.core.tests.db_tests_testapp',
'olympia.core.tests.m2m_testapp',
2018-04-09 17:48:02 +03:00
)
# Make sure the debug toolbar isn't used during the tests.
INSTALLED_APPS = [app for app in INSTALLED_APPS if app != 'debug_toolbar']
MIDDLEWARE = tuple(
middleware
for middleware in MIDDLEWARE
if middleware != 'debug_toolbar.middleware.DebugToolbarMiddleware'
)
2014-12-19 17:58:50 +03:00
INTERNAL_ROUTES_ALLOWED = env('INTERNAL_ROUTES_ALLOWED', default=False)
2011-03-08 22:32:27 +03:00
# See settings.py for documentation:
2012-05-02 21:30:28 +04:00
IN_TEST_SUITE = True
2014-11-17 18:55:02 +03:00
DEBUG = False
# We should default to production mode unless otherwise specified
DEV_MODE = False
2014-11-17 18:55:02 +03:00
# We won't actually send an email.
SEND_REAL_EMAIL = True
SITE_URL = EXTERNAL_SITE_URL = 'http://testserver'
INTERNAL_SITE_URL = 'http://testserver'
Update to Django 1.11 (#8738) Port code and tests to Django 1.11. * Update requirements to Django 1.11.14, remove session_csrf * Use --run-syncdb for migration calls * Remove unused setup-docker script * Port code and tests to Django 1.11. * Remove django-session-csrf everywhere, it got replaced by django's own session csrf protection * Make sure we are calling `client.get()` with `follow=True` where necessary * Replace django.core.urlresolvers import with django.urls * Port custom `IconWidgetRenderer` to Django 1.11 as `IconTypeSelect` * Properly use `_queryset_class` for custom manager classes * Make use of `base_manager_name` on our models `Meta` instead of `use_for_related_objects` on the manager where appropriate. This is required to make translations, soft-delete and more work. See comment on Addons.Meta for more details. * Various fixes along the way to make tests work * Port form errors for locale editing forms to Django 1.11 and make use of form.add_error instead of our own custom process * Mark TestDetailPage.test_304 as xfail for now, somehow Etags seem to be broken for that view during tests but work fine in manual tests * Fix amo.celery, we cannot have any imports at the module-level that may import the ORM * Fix global_settings context processor for 500-server error cases * Fix `changed_data` handling in `AMOModelForm` * Remove `queryset_transform` and implement it directly in amo/models * Fix pagination, Django changed how `num_pages` work and how `count` can be overwritten * Add a new test that tests database transforms as `BaseQuerysetTestCase` * Make sure `legacy_redirect` only appends `?` if there are actual GET parameters * Re-implement `LicenseRadioChoiceInput` as `LicenseRadioSelect` which is now Django 1.11 compatible * Remove our custom `form_kwargs` uplift from Django 1.9 * Correct how we're using `Category.objects.create` and only pass `db_name` instead of `name`, previously this didn't error but now it (luckily!) does * Update `FileSelectWidget` to correctly work with Django 1.11, use custom rendering * Fix decorator ordering for `files.views:browse` * Update database settings to correctly use `TEST` object * Fix `select_related` call in `RatingViewSet` * Remove custom `DateTimeCast` implementation * Fix `addon_factory` calls in `reviewers/tests/test_views.py` to correctly set `is_webextension` * Port our `TranslationDescriptor` and `TranslatedField` to Django 1.11. Also, make sure form error handling works * Add a new test that checks that `default=None` is a valid setting for translation fields * Add a new test that checks that translations from related fields are correctly fetched * Remove possible ambiguity from translations `TestWidget` tests * Make sure that the correct `MEDIA_URL` is set in `TestDownloadsUnlistedVersions` to fix external cdn tests * Remove custom happyforms code. Django cleans text fields now by default * Move translations app to the top of INSTALLED_APPS, add a lengthy comment * Pass additional arguments to contribute_to_class, fix smaller nits * Set MEDIA_URL and STATIC_URL to an external value by default. That's how they're configured in dev, stage and prod anyway. Please note that we didn't replace all our custom form rendering with proper templated widgets yet. This will happening in some follow-up work. Fixes #5271, Fixes #1569
2018-07-11 07:56:54 +03:00
STATIC_URL = '%s/static/' % SITE_URL
MEDIA_URL = '%s/user-media/' % SITE_URL
2011-09-17 01:52:04 +04:00
# Tests run with DEBUG=False but we don't want to have to run collectstatic
# everytime, so reset STATICFILES_STORAGE to the default instead of
# ManifestStaticFilesStorage
STATICFILES_STORAGE = 'django.contrib.staticfiles.storage.StaticFilesStorage'
# Overrides whatever storage you might have put in local settings.
DEFAULT_FILE_STORAGE = 'olympia.amo.utils.SafeStorage'
2012-04-06 00:44:55 +04:00
TASK_USER_ID = 1337
Default to 'and' operator for match queries. Remove slug search, prioritize exact matches more. (#7303) * Default to 'and' operator for match queries. Remove slug search, prioritize exact matches more. References many "component: search" issues. What I tested with a database of all public add-ons: Example searches: tab center redux - should find "Tab Center Redux" while "Tab Mix Plus" is probably second and "Redux DevTools" 4th or so Open Image in New Tab -> should find "Open Image in New Tab" while "Open Bookmarks in New Tab" should be 2nd or 3rd CoinHive -> Finds "Coinhive Blocker", "CoinBlock" (prefix search) and "NoMiners" (description) Privacy -> Finds "Privacy Badger", "Privacy Pass", "Privacy Settings", "Google Privacy" (probably 4th or so) and "Blur" (summary + description + many users). Scores "Ghostery" on the first page but ranks it in the middle firebu -> Finds "Firebug", "Firebug Autocompleter", "Firefinder for Firebug" fireb -> Scores "Fire Drag" first, puts "Firebug" approximately 3rd or so Menu Wizzard -> Finds "Menu Wizard" (fuzzy, typo) first, then "Add-ons Manager Context Menu" apparently because it matches good in the title and has many users Frame Demolition -> Finds "Frame Demolition" Demolition -> Finds only "Frame Demolition", same for "Demolation" (typo) reStyle -> Finds "reStyle" and scores a few add-ons that match on "restore" next since the term is similar MegaUpload DownloadHelper -> finds "MegaUpload DownloadHelper" first, scores "Video DownloadHelper" and "RadpidShare DownloadHelper" next. Doesn't find "Popup Blocker" anymore as currently happening on -prod MegaUpload -> only finds "MegaUpload DownloadHelper" and nothing else No Flash -> Scores "No Flash" first, then depending on users "Download Flash and Video", "YouTube Flash Video Player" and "YouTube Flash Player" (not necessarily in that order) Disable Hello, Pocket & Reader+ -> finds "Disable Hello, Pocket & Reader+" first (yeay!), then scores "Reader", "Disable WebRTC" and "In My Pocket" next similarly to what's happening on -prod currently Not working yet: privacybadger -> "Privacy Badger" -> will probably need some kind of ngram filtering and analyzing (#591) eyes -> 'decentraleyes' -> Not sure this should actually work, will probably need some more analyzing too (#591) Not sure if it's specifically only because of these changes but #3248 is fixed. This potentially fixes #7244, #6891, #6837, #6417, mozilla/addons#359. Not sure if this fixes #mozilla/addons#567 but the results look much more promising and the amount of results doesn't explode here. I only have 2.8k add-ons for testing though so I'm not too sure. And might be relevant to #6137. This is a big step towards #2661, I doubt we can call this fixed though. * Fix name tests * Speed up ES tests, make scoring results more predictable by using only one shard and one replica * Add tests * Only test on new apiv3 based search. * Fixup 'get_results' * Remove debug print * Isort imports * Fix test settings, I'm blind. * Adapt number of shards to what we define in settings_test * Test search for grapple * Remove property filtering in legacy api search, fix tests, fix flake8 * Add comment explaining shard config * Fix tests again, add default platform, fix total counts again. * Try to do some fixture cleanup, let's see... * More test refactoring, make use of dfs-query-then-fetch during tests, allow us to test this later too via a waffle flag. * Don't use waffle flag for legacy search, only for apiv3 searches. * Fix unused import, fix usage of 'params' * Use query-then-fetch in a regular addons-view tool, fix a few more tests to be able to handle the waffle-flag * Create the dfs query then fetch flag only in ESTestCase, delete it properly. More serializer fixes * precache the waffle flag for autocomplete tests too * Minor cleanups * Add docs, fix code style * Add a todo * Fix codestyle
2018-01-30 08:26:30 +03:00
# Make sure we have no replicas and only one shard to allow for impedent
# search scoring
2013-09-07 09:03:53 +04:00
ES_DEFAULT_NUM_REPLICAS = 0
Default to 'and' operator for match queries. Remove slug search, prioritize exact matches more. (#7303) * Default to 'and' operator for match queries. Remove slug search, prioritize exact matches more. References many "component: search" issues. What I tested with a database of all public add-ons: Example searches: tab center redux - should find "Tab Center Redux" while "Tab Mix Plus" is probably second and "Redux DevTools" 4th or so Open Image in New Tab -> should find "Open Image in New Tab" while "Open Bookmarks in New Tab" should be 2nd or 3rd CoinHive -> Finds "Coinhive Blocker", "CoinBlock" (prefix search) and "NoMiners" (description) Privacy -> Finds "Privacy Badger", "Privacy Pass", "Privacy Settings", "Google Privacy" (probably 4th or so) and "Blur" (summary + description + many users). Scores "Ghostery" on the first page but ranks it in the middle firebu -> Finds "Firebug", "Firebug Autocompleter", "Firefinder for Firebug" fireb -> Scores "Fire Drag" first, puts "Firebug" approximately 3rd or so Menu Wizzard -> Finds "Menu Wizard" (fuzzy, typo) first, then "Add-ons Manager Context Menu" apparently because it matches good in the title and has many users Frame Demolition -> Finds "Frame Demolition" Demolition -> Finds only "Frame Demolition", same for "Demolation" (typo) reStyle -> Finds "reStyle" and scores a few add-ons that match on "restore" next since the term is similar MegaUpload DownloadHelper -> finds "MegaUpload DownloadHelper" first, scores "Video DownloadHelper" and "RadpidShare DownloadHelper" next. Doesn't find "Popup Blocker" anymore as currently happening on -prod MegaUpload -> only finds "MegaUpload DownloadHelper" and nothing else No Flash -> Scores "No Flash" first, then depending on users "Download Flash and Video", "YouTube Flash Video Player" and "YouTube Flash Player" (not necessarily in that order) Disable Hello, Pocket & Reader+ -> finds "Disable Hello, Pocket & Reader+" first (yeay!), then scores "Reader", "Disable WebRTC" and "In My Pocket" next similarly to what's happening on -prod currently Not working yet: privacybadger -> "Privacy Badger" -> will probably need some kind of ngram filtering and analyzing (#591) eyes -> 'decentraleyes' -> Not sure this should actually work, will probably need some more analyzing too (#591) Not sure if it's specifically only because of these changes but #3248 is fixed. This potentially fixes #7244, #6891, #6837, #6417, mozilla/addons#359. Not sure if this fixes #mozilla/addons#567 but the results look much more promising and the amount of results doesn't explode here. I only have 2.8k add-ons for testing though so I'm not too sure. And might be relevant to #6137. This is a big step towards #2661, I doubt we can call this fixed though. * Fix name tests * Speed up ES tests, make scoring results more predictable by using only one shard and one replica * Add tests * Only test on new apiv3 based search. * Fixup 'get_results' * Remove debug print * Isort imports * Fix test settings, I'm blind. * Adapt number of shards to what we define in settings_test * Test search for grapple * Remove property filtering in legacy api search, fix tests, fix flake8 * Add comment explaining shard config * Fix tests again, add default platform, fix total counts again. * Try to do some fixture cleanup, let's see... * More test refactoring, make use of dfs-query-then-fetch during tests, allow us to test this later too via a waffle flag. * Don't use waffle flag for legacy search, only for apiv3 searches. * Fix unused import, fix usage of 'params' * Use query-then-fetch in a regular addons-view tool, fix a few more tests to be able to handle the waffle-flag * Create the dfs query then fetch flag only in ESTestCase, delete it properly. More serializer fixes * precache the waffle flag for autocomplete tests too * Minor cleanups * Add docs, fix code style * Add a todo * Fix codestyle
2018-01-30 08:26:30 +03:00
ES_DEFAULT_NUM_SHARDS = 1
# Don't enable the signing by default in tests, many would fail trying to sign
# empty or bad zip files, or try posting to the endpoints. We don't want that.
SIGNING_SERVER = ''
# Disable addon signing for unittests, too many would fail trying to sign
# corrupt/bad zip files. These will be enabled explicitly for unittests.
ENABLE_ADDON_SIGNING = False
2014-12-19 17:58:50 +03:00
2016-06-15 17:15:02 +03:00
# Limit logging in tests.
2020-06-05 14:30:11 +03:00
LOGGING['root']['handlers'] = ['null']
for logger in list(LOGGING['loggers'].keys()):
LOGGING['loggers'][logger]['handlers'] = ['null']
# Need to disable celery logging explicitly. Celery configures its logging
# manually and we don't catch their logger in our default config.
LOGGING['loggers']['celery'] = {
'handlers': ['null'],
'level': logging.DEBUG,
'propagate': False,
2016-06-15 17:15:02 +03:00
}
# To speed tests up, crushing uploaded images is disabled in tests except
# where we explicitly want to test pngcrush.
PNGCRUSH_BIN = '/bin/true'
BASKET_API_KEY = 'testkey'
Upgrade Celery to 4.3 (#12073) Upgrade Celery to 4.3.0 This PR upgrades to Celery 4.3, introduces a small hack because of a regression in celery. This PR does not introduce any helpers to make use of celery workers inside the tests, we delayed that to a later point to unblock the upgrade. This PR does fix some inconsistencies with elasticsearch mocks along the way as they were making some tests fail (see comment logs), the mocks are now applied and rolled back much less than before which makes them easier to reason about and to debug. Notes: * acks_late now doesn't acknowledge in case of failures or timeouts (which is the case for index_addons). We could use task_acks_on_failure_or_timeout but I figured the new behavior should be fine. * any CELERY_ALWAYS_EAGER setting in local_settings.py must be replaced by CELERY_TASK_ALWAYS_EAGER commit log: * Implement proper wait_for_tasks helper, make TestTaskQueued work with celery worker * Revert changes removing threading code from es reindex tests * Fix es reindex test, correct always eager usage. * Small cleanups, fix docs, don't reload session worker. * Ensure docker-compose celery result doesn't use the same database as any potential tests * Monkeypatch for celery regression * Actually use our own app where all tasks are already bound to. This also enables logging * Experiment: Group celery worker tests together with elasticsearch and dont run them multitasked * Remove envlist as it doesn't add anything to our specific setup * Fix es-and-celery marker selection * Remove completely redundant tests * Fix hanging tasks, let's see what travis thinks... I'll follow up with cleanups and much more docs once tests are actually succeeding * Define celery.ping in amo.celery, use redis result backend * Cleanup celery worker implementation a bit. Mimic our usage of elasticsearch specific tests, this avoids various necessary workarounds and environment cleanups because django handles things like settings properly in the first place. * Untangle es mocks a bit, raise error in case of not properly stopped patches. * Re-implement state test by using celery manager to wait for results * Explicitly re-reinitialize the workers event loop and dont reuse anything existing * Remove celery worker thread specific code. * Reset default celery result backend setting
2019-09-18 18:36:01 +03:00
# By default all tests are run in always-eager mode. Use `CeleryWorkerTestCase`
# to start an actual celery worker instead.
CELERY_TASK_ALWAYS_EAGER = True
CELERY_IMPORTS += (
'olympia.amo.tests.test_celery',
'olympia.search.tests.test_commands',
'olympia.devhub.tests.test_tasks',
Upgrade Celery to 4.3 (#12073) Upgrade Celery to 4.3.0 This PR upgrades to Celery 4.3, introduces a small hack because of a regression in celery. This PR does not introduce any helpers to make use of celery workers inside the tests, we delayed that to a later point to unblock the upgrade. This PR does fix some inconsistencies with elasticsearch mocks along the way as they were making some tests fail (see comment logs), the mocks are now applied and rolled back much less than before which makes them easier to reason about and to debug. Notes: * acks_late now doesn't acknowledge in case of failures or timeouts (which is the case for index_addons). We could use task_acks_on_failure_or_timeout but I figured the new behavior should be fine. * any CELERY_ALWAYS_EAGER setting in local_settings.py must be replaced by CELERY_TASK_ALWAYS_EAGER commit log: * Implement proper wait_for_tasks helper, make TestTaskQueued work with celery worker * Revert changes removing threading code from es reindex tests * Fix es reindex test, correct always eager usage. * Small cleanups, fix docs, don't reload session worker. * Ensure docker-compose celery result doesn't use the same database as any potential tests * Monkeypatch for celery regression * Actually use our own app where all tasks are already bound to. This also enables logging * Experiment: Group celery worker tests together with elasticsearch and dont run them multitasked * Remove envlist as it doesn't add anything to our specific setup * Fix es-and-celery marker selection * Remove completely redundant tests * Fix hanging tasks, let's see what travis thinks... I'll follow up with cleanups and much more docs once tests are actually succeeding * Define celery.ping in amo.celery, use redis result backend * Cleanup celery worker implementation a bit. Mimic our usage of elasticsearch specific tests, this avoids various necessary workarounds and environment cleanups because django handles things like settings properly in the first place. * Untangle es mocks a bit, raise error in case of not properly stopped patches. * Re-implement state test by using celery manager to wait for results * Explicitly re-reinitialize the workers event loop and dont reuse anything existing * Remove celery worker thread specific code. * Reset default celery result backend setting
2019-09-18 18:36:01 +03:00
)
CELERY_TASK_ROUTES.update(
{
# Test tasks that will never really be triggered in prod.
'olympia.amo.tests.test_celery.fake_task': {'queue': 'amo'},
'olympia.amo.tests.test_celery.fake_task_with_result': {'queue': 'amo'},
'olympia.amo.tests.test_celery.sleeping_task': {'queue': 'amo'},
'olympia.search.tests.test_commands.dummy_task': {'queue': 'amo'},
'olympia.devhub.tests.test_tasks.fake_task': {'queue': 'amo'},
}
)
# switch cached_db out for just cache sessions to avoid extra db queries
SESSION_ENGINE = 'django.contrib.sessions.backends.cache'
ADDONS_FRONTEND_PROXY_PORT = None
VERIFY_FXA_ACCESS_TOKEN = False
CINDER_API_TOKEN = 'fake-test-token'
CINDER_QUEUE_PREFIX = 'amo-env-'
SOCKET_LABS_TOKEN = 'fake-test-token'
SOCKET_LABS_SERVER_ID = '12345'
SOCKET_LABS_HOST = 'https://fake-socketlabs.com/v1/'