* Remove old update service
* Remove more code/docs associated with services/
* Add some missing pytest.mark.django_db that the autouse fixture was hiding...
* Remove extra import
* Add a comment documenting what the update URL typically looks like
* let our tests run with django3.2
* PositiveAutoField fixes
* versions/test_models.py test fix
* TransField fix
* cache fixes
* use django.utils.html.escape to workaround &# encoding changing to hex
* deactive failing jobs on circleci
* django 2.2.18
* Fix database encoding and exception handling in update service
* Remove leftover ipdb
* Cleaner way to override SERVICES_DATABASE in tests through pytest fixture
* Remove now unused imports
* Upgrade to Elasticsearch 6.x
Because ES 6.x removed mapping types, move update_count and download_count in
separate ES indexes. This requires a full reindex, including stats, but we need
to create an entirely new cluster anyway.
ES 5.x compatibility is kept for now.
Upgrade Celery to 4.3.0
This PR upgrades to Celery 4.3, introduces a small hack because of a regression in celery.
This PR does not introduce any helpers to make use of celery workers inside the tests, we delayed that to a later point to unblock the upgrade.
This PR does fix some inconsistencies with elasticsearch mocks along the way as they were making some tests fail (see comment logs), the mocks are now applied and rolled back much less than before which makes them easier to reason about and to debug.
Notes:
* acks_late now doesn't acknowledge in case of failures or timeouts (which is the case for index_addons). We could use task_acks_on_failure_or_timeout but I figured the new behavior should be fine.
* any CELERY_ALWAYS_EAGER setting in local_settings.py must be replaced by CELERY_TASK_ALWAYS_EAGER
commit log:
* Implement proper wait_for_tasks helper, make TestTaskQueued work with celery worker
* Revert changes removing threading code from es reindex tests
* Fix es reindex test, correct always eager usage.
* Small cleanups, fix docs, don't reload session worker.
* Ensure docker-compose celery result doesn't use the same database as any potential tests
* Monkeypatch for celery regression
* Actually use our own app where all tasks are already bound to. This also enables logging
* Experiment: Group celery worker tests together with elasticsearch and dont run them multitasked
* Remove envlist as it doesn't add anything to our specific setup
* Fix es-and-celery marker selection
* Remove completely redundant tests
* Fix hanging tasks, let's see what travis thinks... I'll follow up with cleanups and much more docs once tests are actually succeeding
* Define celery.ping in amo.celery, use redis result backend
* Cleanup celery worker implementation a bit.
Mimic our usage of elasticsearch specific tests, this avoids various
necessary workarounds and environment cleanups because django handles
things like settings properly in the first place.
* Untangle es mocks a bit, raise error in case of not properly stopped patches.
* Re-implement state test by using celery manager to wait for results
* Explicitly re-reinitialize the workers event loop and dont reuse anything existing
* Remove celery worker thread specific code.
* Reset default celery result backend setting
* Remove all usage of 'six' and as much compat code as I could find.
Cleans up some imports along the way.
Fixes#11728
* Fix typo
* Fix rta related code paths, I actually misread the comment…
* Move ResourceWarning filtering to setup.cfg
Fixes#10420
* Python fixes for the 'es' job
* Remove some legacy api 1.2 cruft, test legacy_api with 1.5 by default
At this point clients using the legacy api are old, clients using the
legacy api < 1.5 are *super* old and not worth supporting.
* Dynamically ignore ResourceWarning under Python 3 for now
* Make stats csv column ordering stable
* Move 'es' job to the python3working section
* Adds a test to test unicode support for latest python-memcached
versions
* Configures caches key prefix so that tests don't conflict with each other
* Remove a few cache.clear calls with `cache.delete` calls that avoids
deleting all memcached entries while other tests are running
* Remove zadmin.memcached as it's crazy dangerous, such power should
only be wielded by ops-folks who wake up at night when pressed
* Fixes waffle helpers to call .flush() to flush appropriate caches
This also fixes how parameters are forwarded to memcached and fix KEY_PREFIX initialization
We are now making sure parameters from `settings.CACHES` are properly forwarded
to the actual cache backend (now memcached). These errors were not seen
since LocMemCache didn't require any parameters.
The `KEY_PREFIX` setting is set to a uuid instead of the xdist-slaveid to
ensure test functions are properly isolated from each other.
`cache.clear` is not called anywhere unless absolutely necessary to reduce
the side-effects of our test-setup.
Fixes#8602
* Unify storage path related settings.
* Moves all storage related settings to settings_base
* Keeps ability to overload base path by setting `NETAPP_STORAGE_ROOT`
* Removes unused PACKAGER_PATH
Fixes#9654
* Sort settings, rename according to their actual path settings
* explicitly set all paths in conftest and document it's usage
* Remove reviewer_attachments folder reference
Fixes#8751
* Add new GIT_FILE_STORAGE_PATH setting
* refactor existing path settings to use os.path.join along the way
* SafeZip validates by default now, `.is_valid` is now a property and
not a method anymore
* Implement new `force_fsync` option to `SafeZip` to force a fsync call
that may be needed for files we require to be absolutely available after
extraction (because we're going to commit them to git)
* Remove nested zip expansion as it's not needed anymore, validation
will be moved to addons-linter and issuing a proper warning
* Add Version.git_hash to point to the latest hash saved.
The git repository management is working almost on an atomic level,
kinda... we are using git worktree
(https://git-scm.com/docs/git-worktree) to create a temporary working
directory for the extraction of a file-object. And once extraction and
comitting is done, we are setting the current HEAD of the respective
branch (listed or unlisted) to that commit hash. That way parallel
extraction works and doesn't need any locking in case someone submits
two versions very quickly after another via the API or so.
Next steps are the creation of a celery-task and management command that
creates necessary repositories in the background, repo-creation
during upload (separate PRs) and reusing FsyncedZipFile in the file-viewer code.
This should fix most of our current Django 2.x and DRF deprecation warnings that we see during test runs.
* Uses `remote_field.model` in amo/utils instead of `remote_field.to`
* Uses `is_anonymous` and `is_authenticated` as properties instead of functions
* Replaces drf `@detail_route` with `@action(detail=True)`
Fixes#9432
* Update pytest-django from 3.3.3 to 3.4.2
* Try not to import models code at the top of the conftest module
* More cleanup, forcefully call django.setup
* Add docs
Only the language tools API used the filesystem cache, and that was
because it had a relatively low usage compared to cache-machine, which
meant everything it cached was quickly evicted. Now that cache-machine
is gone, let's use memcached again for this.
This will simplify the upgrade to Django 1.11 and allow us to keep this
under control, upstream isn't properly maintained.
* Cleanup jingo-minify, keep only parts we are using. Cleanup tests
Also fix and simplify compress_assets to link to the correct /static/ folder.
Originally we hardcoded to look for settings.STATIC_ROOT but that
doesn't suffice since it's set to /site-static/ by default.
We are now using hardcoding the source of our static files instead of
guessing and mistakenly re-using STATIC_ROOT.
This commit also removes verbosity as an argument and simply logs all
errors immediately.
To simplify testing it adds a --force parameter.
To verify this works locally:
* Change settings.STATIC_ROOT to `'/site-static/`'
* Add `/site-static/` to the folders served by nginx (I'll open an issue
on our nginx container to serve this by default too)
* You may have to restart the nginx container after that for nginx to
pick things up
* Remove everything inside site-static folder
* Run `make update_assets`
Fixes#8532
This will simplify the upgrade to Django 1.11 and allow us to keep this
under control, upstream isn't properly maintained.
* Copies jingo-minify into our code
* Removed all parts we aren't using (yui, git integration, `{{ inline_css }}` template tag etc)
* Try to clean up how subprocess calls are being made
* Cleaned up tests
This doesn't change anything regarding how jingo-minify works. It's still using good old `cat` and other ugly things but the code now looks nicer.
See 742297fef9 for more specific details about the cleanup from the original jingo-minify dump.
Fixes#8532
* Implement basket <-> amo newsletter sync.
* Adds basket-client dependency
* Refactors users.notifications to allow for remotely-managed
notification types
* Implement helpers to fetch, subscribe and unsubscribe from newsletters
* Update user profile editing form to fetch and save to basket
* Update accounts/notifications endpoint to fetch and save to basket
* Implement a sync_basket command that fetches basket tokens
asynchronously for all users
All this is behind a waffle flag.
Fixes#7808
* Switch to 100% synchronous basket requests again
* Fix accounts view test to use the conftest configured token
* Fix typos and comments
* Put basket integration behind waffle flag
* Fix typo
* Fix tests
* Remove sync_basket command
* Fix flake8
- Allow filtering by type to only show language packs
- Expose a new current_compatible_version if appversion is passed.
- This property will fetch the latest publicly available version
compatible with the appversion passed.
- Replace caching with a filesystem-based implementation to work
around cache eviction issues.
* Default to 'and' operator for match queries. Remove slug search, prioritize exact matches more.
References many "component: search" issues. What I tested with a
database of all public add-ons:
Example searches:
tab center redux - should find "Tab Center Redux" while "Tab Mix Plus" is probably second and "Redux DevTools" 4th or so
Open Image in New Tab -> should find "Open Image in New Tab" while "Open Bookmarks in New Tab" should be 2nd or 3rd
CoinHive -> Finds "Coinhive Blocker", "CoinBlock" (prefix search) and "NoMiners" (description)
Privacy -> Finds "Privacy Badger", "Privacy Pass", "Privacy Settings", "Google Privacy" (probably 4th or so) and "Blur" (summary + description + many users). Scores "Ghostery" on the first page but ranks it in the middle
firebu -> Finds "Firebug", "Firebug Autocompleter", "Firefinder for Firebug"
fireb -> Scores "Fire Drag" first, puts "Firebug" approximately 3rd or so
Menu Wizzard -> Finds "Menu Wizard" (fuzzy, typo) first, then "Add-ons Manager Context Menu" apparently because it matches good in the title and has many users
Frame Demolition -> Finds "Frame Demolition"
Demolition -> Finds only "Frame Demolition", same for "Demolation" (typo)
reStyle -> Finds "reStyle" and scores a few add-ons that match on "restore" next since the term is similar
MegaUpload DownloadHelper -> finds "MegaUpload DownloadHelper" first, scores "Video DownloadHelper" and "RadpidShare DownloadHelper" next. Doesn't find "Popup Blocker" anymore as currently happening on -prod
MegaUpload -> only finds "MegaUpload DownloadHelper" and nothing else
No Flash -> Scores "No Flash" first, then depending on users "Download Flash and Video", "YouTube Flash Video Player" and "YouTube Flash Player" (not necessarily in that order)
Disable Hello, Pocket & Reader+ -> finds "Disable Hello, Pocket & Reader+" first (yeay!), then scores "Reader", "Disable WebRTC" and "In My Pocket" next similarly to what's happening on -prod currently
Not working yet:
privacybadger -> "Privacy Badger" -> will probably need some kind of ngram filtering and analyzing (#591)
eyes -> 'decentraleyes' -> Not sure this should actually work, will probably need some more analyzing too (#591)
Not sure if it's specifically only because of these changes but #3248 is fixed.
This potentially fixes#7244, #6891, #6837, #6417, mozilla/addons#359.
Not sure if this fixes #mozilla/addons#567 but the results look much more promising and the amount of results doesn't explode here. I only have 2.8k add-ons for testing though so I'm not too sure.
And might be relevant to #6137.
This is a big step towards #2661, I doubt we can call this fixed though.
* Fix name tests
* Speed up ES tests, make scoring results more predictable by using only one shard and one replica
* Add tests
* Only test on new apiv3 based search.
* Fixup 'get_results'
* Remove debug print
* Isort imports
* Fix test settings, I'm blind.
* Adapt number of shards to what we define in settings_test
* Test search for grapple
* Remove property filtering in legacy api search, fix tests, fix flake8
* Add comment explaining shard config
* Fix tests again, add default platform, fix total counts again.
* Try to do some fixture cleanup, let's see...
* More test refactoring, make use of dfs-query-then-fetch during tests, allow us to test this later too via a waffle flag.
* Don't use waffle flag for legacy search, only for apiv3 searches.
* Fix unused import, fix usage of 'params'
* Use query-then-fetch in a regular addons-view tool, fix a few more tests to be able to handle the waffle-flag
* Create the dfs query then fetch flag only in ESTestCase, delete it properly. More serializer fixes
* precache the waffle flag for autocomplete tests too
* Minor cleanups
* Add docs, fix code style
* Add a todo
* Fix codestyle
* Also let the dot in py.test die so that upstream is happy again :)
* Update tox to 2.9.1
* Make sure that our urlconf is reset *after* we set MEDIA_ROOT
* Adapt file extraction handling to new TMP_PATH prefixes
* Less dump_apps hacks since the apps.json file isn't present at the
beginning of any test now thanks to temporary paths for every separate
test
* Rewrite read-only-mode tests to cope with pytest settings fixture
* Remove atexit hack from settings_test
This will probably make things a lot easier for ui-tests as well.
Fixes#7205Fixes#7206
Loads of clean-ups along the way.
Fixes#1573
* Moves all helpers modules to templatetags.jinja_helpers - there are a few things in there that aren't jinja related so there'll be a follow-up to split that later
* Update Jinja2 to 2.9.6
* Add django-jinja 2.3.1
* move some jingo-minify helpers to olympia.lib to avoid any jingo imports
* unified |f and |fe helpers into one |format_html helper that always escapes everything unless the input is already |safe
* Move helpers, add requirements, fix imports.
* Give backend a proper name
* Port f and fe filters from jingo
* Make start porting reviews jinja helpers module, make first test pass
* Get a handful more tests working, get rid of get_env, more and more jingo calls.
* Disable extension matching, render .txt templates too, fixes more stats tests
* Fix csv_header template to produce some reasonable and csv parsable
* Set trim_blocks to be compatible with jingo
* Unify |f and |fe into one |format that simplifies escaping rules. Update tests.
I think having one unified |format filter is beneficial, especially
given that our current escaping rules were kinda complicated and allowed
for seriously dangerous combinations even with autoescape activated.
Now everything passed as an argument to |format is explicitly escaped
unless marked as |safe, there's no way around that anymore.
* Naive first version using |format instead of |f and |fe, might need more review.
* Fix format for unicode strings
* Rename |format to |format_html
* Move nl2br_xhtml to amo.jinja_helpers, use it in all our templates.
* Move datetime filter to our code temporarily, fix mocks, fix recursive error
* Move csrf call to django_jinja csrf_token tag, fix addons templatetags
* Fix |class_selected usage, fix render_with to get registered
* Remove hack from zadmin, fix zadmin templatetags
* Fix user delete photo test, don't rely on proper context updates that happen in order.
* Fix more nl2br tests
* Fix detail page escape, django escapes differently than jinja
* Remove context from edit_review_reply_form
* Fix date created format filter usage
* Remove templates setting user, fix can_review
* Fix collect_add_widget to not inherit the global context
* Remove happy unicode client
* Cleanup can_review, add tests
* Fix tests, django use different html character for double quote escaping
* Switch to firefox latest to check ui-tests
* Cleanup and fix do_truncate
* Remove jingo, copy jingo-minify helpers that depend on jingo to our lib for now
* Rename nl2br_xhtml -> nl2br
* Jinja 2.9 compat for reviews truncate filter
* Don't use TEMPLATE_DEBUG
* Set default value for q, happens on index page.
* Fix jinja2 formatting with newstyle gettext
It took me days to find this and loads of useless refactoring but
apparently these tasks get immediately upgraded to a chord after we
chain them later with the other tasks (after the for-loop).
And, the only requirement for a chord is: do not ignore the results,
never!
Refs #3960
* Make sure index_stats isn't using call_command
* modernize create_subtasks, add support for countdowns, fix queries loading the world...
* half chunk size for stats reindexing
* Refactor stats reindex logging
* Upgrade elasticsearch to 5.4.1 for our docker config
* Make es test setup easier to inherit for our uitests.
* Restore order, make sure to start es before add-on get's created.
* Remove db requirement from es_test fixture
- Middlewares are integrated to existing ones or moved to amo.middlewares
- Logging is moved to core.logger
- All logging calls are modified to use core.logger
- In addition, get/set_user(), get/set_remote_addr() are moved to core
to circumvent import issues.
* Pull in django-waffle 0.10 and a fixed version of django-piston.
Both provide late-initialization support by not calling get_user_model
on import-time.
* Add alias for addons.buttons.js
We now have something like that:
- /
-- src/
--------- olympia/
-------------- {access,accounts,amo,addons,...}
-------------- wsgi.py
-------------- urls.py
-------------- migrations/
-------------- libs/
...
Which generally feels more django-like again and makes porting
our sys.path hacks to django 1.7 and it's app-refactoring
a lot easier.
* Update setup.py to bring it to a more modern format
* Update gitignore
Mathieu Pillard
Andrew Williamson
Christopher Grebs
Mathieu Pillard
pyup.io bot
Mark Striemer
Kris Maglione
Mathieu Agopian