* Replace post-request-task with transaction.on_commit on every delay/apply_async
According to Django docs, transaction.on_commit should:
- only be called once the outer transaction.atomic() has been executed,
guaranteeing that it works even if there are savepoints.
- be discarded on rollback
So it should be a drop-in replacement with the added benefit that it handles
tasks and commands doing transactions outside of the request-response cycle.
* Queue Cinder jobs on a queue per entity type
This changes the filtering the webhook did on the queue, relying on the
job_id to filter jobs we shouldn't be processing, except for jobs in the
special queue for CinderAddonHandledByReviewers.
This allows us to execute decisions from any queue the job might be in,
as long as we know the job_id and that the entity is still an AMO one,
which is useful as jobs might be bounced to a global queue not specific
to AMO.
* chore(users): use url utils forming socketlab urls
* feat(user): add status field to SuppressedEmailVerification model
* feat(): implement suppressed email confirmation tasks
* feat(users): add email_block model + migration
tmp: squash
* feat(): add task to sync blocked emails from socketlabs
Update src/olympia/lib/settings_base.py
Co-authored-by: Andrew Williamson <awilliamson@mozilla.com>
Update src/olympia/users/tasks.py
Co-authored-by: Andrew Williamson <awilliamson@mozilla.com>
Update src/olympia/users/tasks.py
Co-authored-by: Andrew Williamson <awilliamson@mozilla.com>
* tmp: use tempfile to process csv instead of storage
* tmp: rename emailblock to suppressedemail
---------
Co-authored-by: Andrew Williamson <awilliamson@mozilla.com>
* ignore rule F405 per file
* ruff autofix of B007 and B010
* B007 manual fixes
* B026 fixes
* B023 fixes
* updated ruff ignore list
* black formatting & rework decorators
* optimize iterating on dicts where we don't need the keys or values
* Refactor search code into a simpler set of modules
- `olympia.search` should contain everything to interact with ES
- `olympia.addons.indexers` continue to contain the mapping for `Addon`
* Disable ES verification step
* Replace client.login(email='...') with client.force_login(user) in tests
This allows us to remove the fake auth backend hack used for tests
* Test fixes
* Remove code dedicated domain CDN - download xpis through the main domain
- Add comments to ensure we don't change the download.file pattern without
checking with Fenix
* Use DB sessions; store refresh tokens in session; use for frontend auth too
* add clearsessions to cron
* return ERROR_SIGNATURE_EXPIRED for every scenerio that can be expiring sessions
* fix tests after default_fxa_login_url removed
* Return ERROR_AUTHENTICATION_EXPIRED for SessionIDAuthentication problems
* Bump django from 3.2.10 to 3.2.11 in /requirements
Bumps [django](https://github.com/django/django) from 3.2.10 to 3.2.11.
- [Release notes](https://github.com/django/django/releases)
- [Commits](https://github.com/django/django/compare/3.2.10...3.2.11)
---
updated-dependencies:
- dependency-name: django
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
* use str for paths everywhere; make our Storage subclass safer
* enhance SafeStorage to be used per user_media path
* test refactoring
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* Replace legacy assets cachebusting by staticfiles's ManifestStaticFilesStorage from django
* Run tests without ManifestStaticFilesStorage
* Fix tests
* Reformatting
* It's fine, since collectstatic also copies the original
* Register static directly
* Reformat comment
* let our tests run with django3.2
* PositiveAutoField fixes
* versions/test_models.py test fix
* TransField fix
* cache fixes
* use django.utils.html.escape to workaround &# encoding changing to hex
* deactive failing jobs on circleci
* django 2.2.18
* soft delete AddonUser
* use _prefetched_object_cache in get_queryset
* Document & test FilterableManyToManyDescriptor
* Add test for basketsync with AddonUser hard delete
* Remove obsolete code associated with repack_themes_for_69 task
* Remove obsolete content_approve_migrated_themes task & command
* Explicitly set a route for each celery task
* Remove obsolete delete_armagaddon_ratings_for_addons task
* Let `default` be selected as a celery queue in explicit routing
* Remove unused imports
* Also include this test task
* Update settings_tests instead of settings_base for tests stuff
Upgrade Celery to 4.3.0
This PR upgrades to Celery 4.3, introduces a small hack because of a regression in celery.
This PR does not introduce any helpers to make use of celery workers inside the tests, we delayed that to a later point to unblock the upgrade.
This PR does fix some inconsistencies with elasticsearch mocks along the way as they were making some tests fail (see comment logs), the mocks are now applied and rolled back much less than before which makes them easier to reason about and to debug.
Notes:
* acks_late now doesn't acknowledge in case of failures or timeouts (which is the case for index_addons). We could use task_acks_on_failure_or_timeout but I figured the new behavior should be fine.
* any CELERY_ALWAYS_EAGER setting in local_settings.py must be replaced by CELERY_TASK_ALWAYS_EAGER
commit log:
* Implement proper wait_for_tasks helper, make TestTaskQueued work with celery worker
* Revert changes removing threading code from es reindex tests
* Fix es reindex test, correct always eager usage.
* Small cleanups, fix docs, don't reload session worker.
* Ensure docker-compose celery result doesn't use the same database as any potential tests
* Monkeypatch for celery regression
* Actually use our own app where all tasks are already bound to. This also enables logging
* Experiment: Group celery worker tests together with elasticsearch and dont run them multitasked
* Remove envlist as it doesn't add anything to our specific setup
* Fix es-and-celery marker selection
* Remove completely redundant tests
* Fix hanging tasks, let's see what travis thinks... I'll follow up with cleanups and much more docs once tests are actually succeeding
* Define celery.ping in amo.celery, use redis result backend
* Cleanup celery worker implementation a bit.
Mimic our usage of elasticsearch specific tests, this avoids various
necessary workarounds and environment cleanups because django handles
things like settings properly in the first place.
* Untangle es mocks a bit, raise error in case of not properly stopped patches.
* Re-implement state test by using celery manager to wait for results
* Explicitly re-reinitialize the workers event loop and dont reuse anything existing
* Remove celery worker thread specific code.
* Reset default celery result backend setting
* Add new olympia.core.languages module as a replacement for supported
languages
* Implement new ApplicationsVersions.get_latest_application_version
helper
Fixes#9516Fixes#8850 along the way, removes the need for #8779 and unblocks #2739
* Adds a test to test unicode support for latest python-memcached
versions
* Configures caches key prefix so that tests don't conflict with each other
* Remove a few cache.clear calls with `cache.delete` calls that avoids
deleting all memcached entries while other tests are running
* Remove zadmin.memcached as it's crazy dangerous, such power should
only be wielded by ops-folks who wake up at night when pressed
* Fixes waffle helpers to call .flush() to flush appropriate caches
This also fixes how parameters are forwarded to memcached and fix KEY_PREFIX initialization
We are now making sure parameters from `settings.CACHES` are properly forwarded
to the actual cache backend (now memcached). These errors were not seen
since LocMemCache didn't require any parameters.
The `KEY_PREFIX` setting is set to a uuid instead of the xdist-slaveid to
ensure test functions are properly isolated from each other.
`cache.clear` is not called anywhere unless absolutely necessary to reduce
the side-effects of our test-setup.
Fixes#8602
Only the language tools API used the filesystem cache, and that was
because it had a relatively low usage compared to cache-machine, which
meant everything it cached was quickly evicted. Now that cache-machine
is gone, let's use memcached again for this.
* Integrate git-release into raven messages.
This renames a few raven related settings that aren't used by the
official raven package but only set in our base templates for Raven.js
to pick them up.
This also removes a few unused settings variables.
Fixes#8851
* Fetch git commit from version.json as well.
Port code and tests to Django 1.11.
* Update requirements to Django 1.11.14, remove session_csrf
* Use --run-syncdb for migration calls
* Remove unused setup-docker script
* Port code and tests to Django 1.11.
* Remove django-session-csrf everywhere, it got replaced by django's own
session csrf protection
* Make sure we are calling `client.get()` with `follow=True` where necessary
* Replace django.core.urlresolvers import with django.urls
* Port custom `IconWidgetRenderer` to Django 1.11 as `IconTypeSelect`
* Properly use `_queryset_class` for custom manager classes
* Make use of `base_manager_name` on our models `Meta` instead of `use_for_related_objects`
on the manager where appropriate. This is required to make translations, soft-delete
and more work. See comment on Addons.Meta for more details.
* Various fixes along the way to make tests work
* Port form errors for locale editing forms to Django 1.11 and make use of
form.add_error instead of our own custom process
* Mark TestDetailPage.test_304 as xfail for now, somehow Etags seem to be broken
for that view during tests but work fine in manual tests
* Fix amo.celery, we cannot have any imports at the module-level that may
import the ORM
* Fix global_settings context processor for 500-server error cases
* Fix `changed_data` handling in `AMOModelForm`
* Remove `queryset_transform` and implement it directly in amo/models
* Fix pagination, Django changed how `num_pages` work and how `count` can be overwritten
* Add a new test that tests database transforms as `BaseQuerysetTestCase`
* Make sure `legacy_redirect` only appends `?` if there are actual GET parameters
* Re-implement `LicenseRadioChoiceInput` as `LicenseRadioSelect` which is now Django 1.11 compatible
* Remove our custom `form_kwargs` uplift from Django 1.9
* Correct how we're using `Category.objects.create` and only pass `db_name` instead of `name`, previously
this didn't error but now it (luckily!) does
* Update `FileSelectWidget` to correctly work with Django 1.11, use custom rendering
* Fix decorator ordering for `files.views:browse`
* Update database settings to correctly use `TEST` object
* Fix `select_related` call in `RatingViewSet`
* Remove custom `DateTimeCast` implementation
* Fix `addon_factory` calls in `reviewers/tests/test_views.py` to correctly set
`is_webextension`
* Port our `TranslationDescriptor` and `TranslatedField` to Django 1.11. Also, make sure form error handling works
* Add a new test that checks that `default=None` is a valid setting for translation fields
* Add a new test that checks that translations from related fields are correctly fetched
* Remove possible ambiguity from translations `TestWidget` tests
* Make sure that the correct `MEDIA_URL` is set in `TestDownloadsUnlistedVersions` to fix
external cdn tests
* Remove custom happyforms code. Django cleans text fields now by default
* Move translations app to the top of INSTALLED_APPS, add a lengthy comment
* Pass additional arguments to contribute_to_class, fix smaller nits
* Set MEDIA_URL and STATIC_URL to an external value by default. That's how they're configured in dev, stage and prod anyway.
Please note that we didn't replace all our custom form rendering with proper templated widgets yet. This will
happening in some follow-up work.
Fixes#5271, Fixes#1569
Closes#8451 (Doesn't necessary fix it, we don't want a unified logging-level but this PR makes logging much clearer.)
* Don't use verbose mode for tox
* Refactor logging settings
* Fix services logging, correct dev and stage settings, update docs
* Update logging
* Be vigilent about disabling all logging
* Remove MOZLOG_NAME from stage settings
* Implement basket <-> amo newsletter sync.
* Adds basket-client dependency
* Refactors users.notifications to allow for remotely-managed
notification types
* Implement helpers to fetch, subscribe and unsubscribe from newsletters
* Update user profile editing form to fetch and save to basket
* Update accounts/notifications endpoint to fetch and save to basket
* Implement a sync_basket command that fetches basket tokens
asynchronously for all users
All this is behind a waffle flag.
Fixes#7808
* Switch to 100% synchronous basket requests again
* Fix accounts view test to use the conftest configured token
* Fix typos and comments
* Put basket integration behind waffle flag
* Fix typo
* Fix tests
* Remove sync_basket command
* Fix flake8
* Implement cache request tracking, add zadmin config tests.
Implements CacheStatTracker as a backend for tests to allow us to use
`assert_cache_requests` in tests.
* implement a `cached` helper and removes caching.{cached,cached_with} instances.
* change from caching backends back to django cache backends
* Move cache_helpers -> cache_nuggets
* Fix CacheStatTracker to not remove OPTIONS values
- Allow filtering by type to only show language packs
- Expose a new current_compatible_version if appversion is passed.
- This property will fetch the latest publicly available version
compatible with the appversion passed.
- Replace caching with a filesystem-based implementation to work
around cache eviction issues.
Icons and previews for both themes and non-themes should go through pngcrush.
For the existing content, the command `crush_images_for_top_addons` focuses
on the content displayed on landing pages only and should only be run once.
* Default to 'and' operator for match queries. Remove slug search, prioritize exact matches more.
References many "component: search" issues. What I tested with a
database of all public add-ons:
Example searches:
tab center redux - should find "Tab Center Redux" while "Tab Mix Plus" is probably second and "Redux DevTools" 4th or so
Open Image in New Tab -> should find "Open Image in New Tab" while "Open Bookmarks in New Tab" should be 2nd or 3rd
CoinHive -> Finds "Coinhive Blocker", "CoinBlock" (prefix search) and "NoMiners" (description)
Privacy -> Finds "Privacy Badger", "Privacy Pass", "Privacy Settings", "Google Privacy" (probably 4th or so) and "Blur" (summary + description + many users). Scores "Ghostery" on the first page but ranks it in the middle
firebu -> Finds "Firebug", "Firebug Autocompleter", "Firefinder for Firebug"
fireb -> Scores "Fire Drag" first, puts "Firebug" approximately 3rd or so
Menu Wizzard -> Finds "Menu Wizard" (fuzzy, typo) first, then "Add-ons Manager Context Menu" apparently because it matches good in the title and has many users
Frame Demolition -> Finds "Frame Demolition"
Demolition -> Finds only "Frame Demolition", same for "Demolation" (typo)
reStyle -> Finds "reStyle" and scores a few add-ons that match on "restore" next since the term is similar
MegaUpload DownloadHelper -> finds "MegaUpload DownloadHelper" first, scores "Video DownloadHelper" and "RadpidShare DownloadHelper" next. Doesn't find "Popup Blocker" anymore as currently happening on -prod
MegaUpload -> only finds "MegaUpload DownloadHelper" and nothing else
No Flash -> Scores "No Flash" first, then depending on users "Download Flash and Video", "YouTube Flash Video Player" and "YouTube Flash Player" (not necessarily in that order)
Disable Hello, Pocket & Reader+ -> finds "Disable Hello, Pocket & Reader+" first (yeay!), then scores "Reader", "Disable WebRTC" and "In My Pocket" next similarly to what's happening on -prod currently
Not working yet:
privacybadger -> "Privacy Badger" -> will probably need some kind of ngram filtering and analyzing (#591)
eyes -> 'decentraleyes' -> Not sure this should actually work, will probably need some more analyzing too (#591)
Not sure if it's specifically only because of these changes but #3248 is fixed.
This potentially fixes#7244, #6891, #6837, #6417, mozilla/addons#359.
Not sure if this fixes #mozilla/addons#567 but the results look much more promising and the amount of results doesn't explode here. I only have 2.8k add-ons for testing though so I'm not too sure.
And might be relevant to #6137.
This is a big step towards #2661, I doubt we can call this fixed though.
* Fix name tests
* Speed up ES tests, make scoring results more predictable by using only one shard and one replica
* Add tests
* Only test on new apiv3 based search.
* Fixup 'get_results'
* Remove debug print
* Isort imports
* Fix test settings, I'm blind.
* Adapt number of shards to what we define in settings_test
* Test search for grapple
* Remove property filtering in legacy api search, fix tests, fix flake8
* Add comment explaining shard config
* Fix tests again, add default platform, fix total counts again.
* Try to do some fixture cleanup, let's see...
* More test refactoring, make use of dfs-query-then-fetch during tests, allow us to test this later too via a waffle flag.
* Don't use waffle flag for legacy search, only for apiv3 searches.
* Fix unused import, fix usage of 'params'
* Use query-then-fetch in a regular addons-view tool, fix a few more tests to be able to handle the waffle-flag
* Create the dfs query then fetch flag only in ESTestCase, delete it properly. More serializer fixes
* precache the waffle flag for autocomplete tests too
* Minor cleanups
* Add docs, fix code style
* Add a todo
* Fix codestyle
* Also let the dot in py.test die so that upstream is happy again :)
* Update tox to 2.9.1
* Make sure that our urlconf is reset *after* we set MEDIA_ROOT
* Adapt file extraction handling to new TMP_PATH prefixes
* Less dump_apps hacks since the apps.json file isn't present at the
beginning of any test now thanks to temporary paths for every separate
test
* Rewrite read-only-mode tests to cope with pytest settings fixture
* Remove atexit hack from settings_test
This will probably make things a lot easier for ui-tests as well.
Fixes#7205Fixes#7206
Fixes#6640Fixes#6866
This pull request adds support for our new Autograph signing server, adds it to our dockerized services and adds tests that now actually call our signing server.
* Add a waffle-switch for calls to our new autograph signing server
* Add the autograph signing server to docker-compose
* Add the autograph signing server to our travis setup
* avoid us specifying the signing server settings everywhere in the code, one single source of truth
* more use of `responses` instead of mocking, leaning towards
the future where we can actually hit a signing server instead
of a mock
* disallow http requests generically, unless configured otherwise
* Add migration for waffle flag
* Update celery, config and dependencies to 4.x
* Remove usage of TaskSet, replace with group(), update `ampq`
* Enable redis on travis for better celery 4.x testing
* Fix lib/celery_tests
* Correct test testing email sending retrying on errors
* Fix celery subtask mocking errors.
* Fix celery broker url env variable
* Integrate watchdog for celery autoreload
* Ensure dependencies are updated for worker and web container.
* Restart all services after the dependency update
* Remove explicit celery loader environment variable.
* Restart all services in web and worker after running 'update_docker'
* Increase sleep time a bit
This reverts commit 541688d5f2.
This caused some breakage on dev. We suspect it's because newrelic
hooks into the celery task launching code but does not support
celery 4.x yet.
* Update celery config to seamlessly upgrade to 4.x
* Update celery initialization for new configs and 4.x
* Update dependencies to celery 4.x
* Fix flake8
* Fix typo
* Remove usage of TaskSet, replace with group(), update `ampq`
* Enable redis on travis for better celery 4.x testing
* Fix more test setting names
* Fix lib/celery_tests
* Correct test testing email sending retrying on errors
* Fix celery subtask mocking errors.
* Fix new line
* Fix celery broker url env variable
* Integrate watchdog for celery autoreload
* Try something weird... ui-tests
* do we need extra hosts?
* more sleep?
* Try logging, try docker run to avoid port collisions
* More debugging
* Cleanup again
* Fix logs?
* Ensure dependencies are updated for worker and web container.
* Remove not functioning log saving for onw
* Restart all services after the dependency update
* Remove HIDDEN_LANGUAGES, sync with .po files and product-details.
Fixes#6534, Fixes#6535
* Remove test referencing cy language, imho it doesn't make sense to have languages lying around in the code that aren't in product-details anyway, just adds another layer of writing things down needlessly
* Add comment about bidi languages
* Add test
* Remove fa-IR from languages, we never supported it in the past.
* Remove custom locales from test settings.
* Make sure we extract and analyze en-gb as english correctly.
Fixes#6156
* Add more tests verifying search behavior
* Actually run search from multiple locales
* Fix test settings to include en-GB and Spanish locales, use more unicode
* Fix flake8
* Fix fake language config
Kevin Meinhardt
Mathieu Pillard
Andrew Williamson
Bob Silverberg
Lisa Chan
William Durand
Christopher Grebs
Mathieu Pillard