In case a developer is using Docker Desktop for Windows then
the containers don't prepare /etc/hosts file. Additionally,
elasticsearch faces OOM errors so added a Gotcha for this.
Upgrade Celery to 4.3.0
This PR upgrades to Celery 4.3, introduces a small hack because of a regression in celery.
This PR does not introduce any helpers to make use of celery workers inside the tests, we delayed that to a later point to unblock the upgrade.
This PR does fix some inconsistencies with elasticsearch mocks along the way as they were making some tests fail (see comment logs), the mocks are now applied and rolled back much less than before which makes them easier to reason about and to debug.
Notes:
* acks_late now doesn't acknowledge in case of failures or timeouts (which is the case for index_addons). We could use task_acks_on_failure_or_timeout but I figured the new behavior should be fine.
* any CELERY_ALWAYS_EAGER setting in local_settings.py must be replaced by CELERY_TASK_ALWAYS_EAGER
commit log:
* Implement proper wait_for_tasks helper, make TestTaskQueued work with celery worker
* Revert changes removing threading code from es reindex tests
* Fix es reindex test, correct always eager usage.
* Small cleanups, fix docs, don't reload session worker.
* Ensure docker-compose celery result doesn't use the same database as any potential tests
* Monkeypatch for celery regression
* Actually use our own app where all tasks are already bound to. This also enables logging
* Experiment: Group celery worker tests together with elasticsearch and dont run them multitasked
* Remove envlist as it doesn't add anything to our specific setup
* Fix es-and-celery marker selection
* Remove completely redundant tests
* Fix hanging tasks, let's see what travis thinks... I'll follow up with cleanups and much more docs once tests are actually succeeding
* Define celery.ping in amo.celery, use redis result backend
* Cleanup celery worker implementation a bit.
Mimic our usage of elasticsearch specific tests, this avoids various
necessary workarounds and environment cleanups because django handles
things like settings properly in the first place.
* Untangle es mocks a bit, raise error in case of not properly stopped patches.
* Re-implement state test by using celery manager to wait for results
* Explicitly re-reinitialize the workers event loop and dont reuse anything existing
* Remove celery worker thread specific code.
* Reset default celery result backend setting
* move all the make commands that operate on a single container to Makefile-docker
* add default user to Dockerfile
* revert the PYTHON_COMMAND var change
Most of this work landed already in
6a8fc341da and
91605a4c1f so this is just a follow-up to
remove all other traces.
The removed comment is no longer true, Django received all the necessary
fixes for LocMemCache. On the other hand, LoMemCache is no longer used
for a regular local setup because we are using memcached and try to be
as close to production as possible so this won't change anytime
soon.
* Also let the dot in py.test die so that upstream is happy again :)
* Update tox to 2.9.1
* Make sure that our urlconf is reset *after* we set MEDIA_ROOT
* Adapt file extraction handling to new TMP_PATH prefixes
* Less dump_apps hacks since the apps.json file isn't present at the
beginning of any test now thanks to temporary paths for every separate
test
* Rewrite read-only-mode tests to cope with pytest settings fixture
* Remove atexit hack from settings_test
This will probably make things a lot easier for ui-tests as well.
Fixes#7205Fixes#7206
Andrew Williamson
Lisa Chan
Shashwat S
Mathieu Pillard
mirefly
yatinmaan
Christopher Grebs
Mac Chaffee
Apoorva Pandey
Stuart Colville
ka7
Matthew Riley MacPherson
Mark Striemer