Upgrade Celery to 4.3.0
This PR upgrades to Celery 4.3, introduces a small hack because of a regression in celery.
This PR does not introduce any helpers to make use of celery workers inside the tests, we delayed that to a later point to unblock the upgrade.
This PR does fix some inconsistencies with elasticsearch mocks along the way as they were making some tests fail (see comment logs), the mocks are now applied and rolled back much less than before which makes them easier to reason about and to debug.
Notes:
* acks_late now doesn't acknowledge in case of failures or timeouts (which is the case for index_addons). We could use task_acks_on_failure_or_timeout but I figured the new behavior should be fine.
* any CELERY_ALWAYS_EAGER setting in local_settings.py must be replaced by CELERY_TASK_ALWAYS_EAGER
commit log:
* Implement proper wait_for_tasks helper, make TestTaskQueued work with celery worker
* Revert changes removing threading code from es reindex tests
* Fix es reindex test, correct always eager usage.
* Small cleanups, fix docs, don't reload session worker.
* Ensure docker-compose celery result doesn't use the same database as any potential tests
* Monkeypatch for celery regression
* Actually use our own app where all tasks are already bound to. This also enables logging
* Experiment: Group celery worker tests together with elasticsearch and dont run them multitasked
* Remove envlist as it doesn't add anything to our specific setup
* Fix es-and-celery marker selection
* Remove completely redundant tests
* Fix hanging tasks, let's see what travis thinks... I'll follow up with cleanups and much more docs once tests are actually succeeding
* Define celery.ping in amo.celery, use redis result backend
* Cleanup celery worker implementation a bit.
Mimic our usage of elasticsearch specific tests, this avoids various
necessary workarounds and environment cleanups because django handles
things like settings properly in the first place.
* Untangle es mocks a bit, raise error in case of not properly stopped patches.
* Re-implement state test by using celery manager to wait for results
* Explicitly re-reinitialize the workers event loop and dont reuse anything existing
* Remove celery worker thread specific code.
* Reset default celery result backend setting
* Remove all usage of 'six' and as much compat code as I could find.
Cleans up some imports along the way.
Fixes#11728
* Fix typo
* Fix rta related code paths, I actually misread the comment…
* Move ResourceWarning filtering to setup.cfg
* Update pytest from 3.7.4 to 3.8.1
* Update pytest from 3.7.4 to 3.8.1
* Update pytest from 3.7.4 to 3.8.1
* Ignore csp.utils deprecation warnings
* Ignore command outpout for static assets test, the 'assets' job is running the collecting manually too
* Remove pytest from prod_common
* Make requirements/uitests depend on tests.txt, remove pytest-cache
* Add isort to our codebase.
This is the first draft, please feel free to discuss this but I'm
generally open to change that style later on.
I just want to start making more and more of our code-style a rule
instead of something we have to take care of during code reviews.
* Ignore safe_xml file
* Don't change safe_lxml_etree
* Fix import unifification going bonkers, I don't blame isort though, this was creepy.
* Fix signin.test_views
* More flake8
* Use style number 4, hanging grid
* django group only contains django now, move above 3rd party group
* Remove pytest-isort for now to avoid isort being imported during tests.
isort has a super bad module that sets the defaultsystemencoding to
utf-8 and breaks our tests that ensure that our settings are *never*
unicode strings. If the default system encoding is set to utf-8 these
tests just work since the bytestring will be implicitly encoded to utf-8
instead of ascii which is the python2 default.
https://github.com/timothycrosley/isort/blob/develop/isort/pie_slice.py#L145
* Remove instafail, show shorter tracebacks
* Cache .tox, node_modules and pip
* Upgrade pip and wheel to make sure we're using cached wheel packages
* Fail the whole suite if one build already failed. This should give faster feedback for PRs
* Pull in django-waffle 0.10 and a fixed version of django-piston.
Both provide late-initialization support by not calling get_user_model
on import-time.
* Add alias for addons.buttons.js
* Update installed apps, fix grouping and naming.
* Fix app-names in settings
* re-add startup import to manage.py for now
Various more import fixes, all import rewrites are now done.
This unfortunately still failes because of some circular imports
depending on django not being setup early enough.
William Durand
Andrew Williamson
Christopher Grebs
pyup.io bot
Mathieu Pillard
Benjamin Forehand Jr
Andreas Wagner
Mathieu Agopian