Fixes#8751
* Add new GIT_FILE_STORAGE_PATH setting
* refactor existing path settings to use os.path.join along the way
* SafeZip validates by default now, `.is_valid` is now a property and
not a method anymore
* Implement new `force_fsync` option to `SafeZip` to force a fsync call
that may be needed for files we require to be absolutely available after
extraction (because we're going to commit them to git)
* Remove nested zip expansion as it's not needed anymore, validation
will be moved to addons-linter and issuing a proper warning
* Add Version.git_hash to point to the latest hash saved.
The git repository management is working almost on an atomic level,
kinda... we are using git worktree
(https://git-scm.com/docs/git-worktree) to create a temporary working
directory for the extraction of a file-object. And once extraction and
comitting is done, we are setting the current HEAD of the respective
branch (listed or unlisted) to that commit hash. That way parallel
extraction works and doesn't need any locking in case someone submits
two versions very quickly after another via the API or so.
Next steps are the creation of a celery-task and management command that
creates necessary repositories in the background, repo-creation
during upload (separate PRs) and reusing FsyncedZipFile in the file-viewer code.
This should fix most of our current Django 2.x and DRF deprecation warnings that we see during test runs.
* Uses `remote_field.model` in amo/utils instead of `remote_field.to`
* Uses `is_anonymous` and `is_authenticated` as properties instead of functions
* Replaces drf `@detail_route` with `@action(detail=True)`
Fixes#9432
* Update pytest-django from 3.3.3 to 3.4.2
* Try not to import models code at the top of the conftest module
* More cleanup, forcefully call django.setup
* Add docs
Only the language tools API used the filesystem cache, and that was
because it had a relatively low usage compared to cache-machine, which
meant everything it cached was quickly evicted. Now that cache-machine
is gone, let's use memcached again for this.
This will simplify the upgrade to Django 1.11 and allow us to keep this
under control, upstream isn't properly maintained.
* Cleanup jingo-minify, keep only parts we are using. Cleanup tests
Also fix and simplify compress_assets to link to the correct /static/ folder.
Originally we hardcoded to look for settings.STATIC_ROOT but that
doesn't suffice since it's set to /site-static/ by default.
We are now using hardcoding the source of our static files instead of
guessing and mistakenly re-using STATIC_ROOT.
This commit also removes verbosity as an argument and simply logs all
errors immediately.
To simplify testing it adds a --force parameter.
To verify this works locally:
* Change settings.STATIC_ROOT to `'/site-static/`'
* Add `/site-static/` to the folders served by nginx (I'll open an issue
on our nginx container to serve this by default too)
* You may have to restart the nginx container after that for nginx to
pick things up
* Remove everything inside site-static folder
* Run `make update_assets`
Fixes#8532
This will simplify the upgrade to Django 1.11 and allow us to keep this
under control, upstream isn't properly maintained.
* Copies jingo-minify into our code
* Removed all parts we aren't using (yui, git integration, `{{ inline_css }}` template tag etc)
* Try to clean up how subprocess calls are being made
* Cleaned up tests
This doesn't change anything regarding how jingo-minify works. It's still using good old `cat` and other ugly things but the code now looks nicer.
See 742297fef9 for more specific details about the cleanup from the original jingo-minify dump.
Fixes#8532
* Implement basket <-> amo newsletter sync.
* Adds basket-client dependency
* Refactors users.notifications to allow for remotely-managed
notification types
* Implement helpers to fetch, subscribe and unsubscribe from newsletters
* Update user profile editing form to fetch and save to basket
* Update accounts/notifications endpoint to fetch and save to basket
* Implement a sync_basket command that fetches basket tokens
asynchronously for all users
All this is behind a waffle flag.
Fixes#7808
* Switch to 100% synchronous basket requests again
* Fix accounts view test to use the conftest configured token
* Fix typos and comments
* Put basket integration behind waffle flag
* Fix typo
* Fix tests
* Remove sync_basket command
* Fix flake8
- Allow filtering by type to only show language packs
- Expose a new current_compatible_version if appversion is passed.
- This property will fetch the latest publicly available version
compatible with the appversion passed.
- Replace caching with a filesystem-based implementation to work
around cache eviction issues.
* Default to 'and' operator for match queries. Remove slug search, prioritize exact matches more.
References many "component: search" issues. What I tested with a
database of all public add-ons:
Example searches:
tab center redux - should find "Tab Center Redux" while "Tab Mix Plus" is probably second and "Redux DevTools" 4th or so
Open Image in New Tab -> should find "Open Image in New Tab" while "Open Bookmarks in New Tab" should be 2nd or 3rd
CoinHive -> Finds "Coinhive Blocker", "CoinBlock" (prefix search) and "NoMiners" (description)
Privacy -> Finds "Privacy Badger", "Privacy Pass", "Privacy Settings", "Google Privacy" (probably 4th or so) and "Blur" (summary + description + many users). Scores "Ghostery" on the first page but ranks it in the middle
firebu -> Finds "Firebug", "Firebug Autocompleter", "Firefinder for Firebug"
fireb -> Scores "Fire Drag" first, puts "Firebug" approximately 3rd or so
Menu Wizzard -> Finds "Menu Wizard" (fuzzy, typo) first, then "Add-ons Manager Context Menu" apparently because it matches good in the title and has many users
Frame Demolition -> Finds "Frame Demolition"
Demolition -> Finds only "Frame Demolition", same for "Demolation" (typo)
reStyle -> Finds "reStyle" and scores a few add-ons that match on "restore" next since the term is similar
MegaUpload DownloadHelper -> finds "MegaUpload DownloadHelper" first, scores "Video DownloadHelper" and "RadpidShare DownloadHelper" next. Doesn't find "Popup Blocker" anymore as currently happening on -prod
MegaUpload -> only finds "MegaUpload DownloadHelper" and nothing else
No Flash -> Scores "No Flash" first, then depending on users "Download Flash and Video", "YouTube Flash Video Player" and "YouTube Flash Player" (not necessarily in that order)
Disable Hello, Pocket & Reader+ -> finds "Disable Hello, Pocket & Reader+" first (yeay!), then scores "Reader", "Disable WebRTC" and "In My Pocket" next similarly to what's happening on -prod currently
Not working yet:
privacybadger -> "Privacy Badger" -> will probably need some kind of ngram filtering and analyzing (#591)
eyes -> 'decentraleyes' -> Not sure this should actually work, will probably need some more analyzing too (#591)
Not sure if it's specifically only because of these changes but #3248 is fixed.
This potentially fixes#7244, #6891, #6837, #6417, mozilla/addons#359.
Not sure if this fixes #mozilla/addons#567 but the results look much more promising and the amount of results doesn't explode here. I only have 2.8k add-ons for testing though so I'm not too sure.
And might be relevant to #6137.
This is a big step towards #2661, I doubt we can call this fixed though.
* Fix name tests
* Speed up ES tests, make scoring results more predictable by using only one shard and one replica
* Add tests
* Only test on new apiv3 based search.
* Fixup 'get_results'
* Remove debug print
* Isort imports
* Fix test settings, I'm blind.
* Adapt number of shards to what we define in settings_test
* Test search for grapple
* Remove property filtering in legacy api search, fix tests, fix flake8
* Add comment explaining shard config
* Fix tests again, add default platform, fix total counts again.
* Try to do some fixture cleanup, let's see...
* More test refactoring, make use of dfs-query-then-fetch during tests, allow us to test this later too via a waffle flag.
* Don't use waffle flag for legacy search, only for apiv3 searches.
* Fix unused import, fix usage of 'params'
* Use query-then-fetch in a regular addons-view tool, fix a few more tests to be able to handle the waffle-flag
* Create the dfs query then fetch flag only in ESTestCase, delete it properly. More serializer fixes
* precache the waffle flag for autocomplete tests too
* Minor cleanups
* Add docs, fix code style
* Add a todo
* Fix codestyle
* Also let the dot in py.test die so that upstream is happy again :)
* Update tox to 2.9.1
* Make sure that our urlconf is reset *after* we set MEDIA_ROOT
* Adapt file extraction handling to new TMP_PATH prefixes
* Less dump_apps hacks since the apps.json file isn't present at the
beginning of any test now thanks to temporary paths for every separate
test
* Rewrite read-only-mode tests to cope with pytest settings fixture
* Remove atexit hack from settings_test
This will probably make things a lot easier for ui-tests as well.
Fixes#7205Fixes#7206
Loads of clean-ups along the way.
Fixes#1573
* Moves all helpers modules to templatetags.jinja_helpers - there are a few things in there that aren't jinja related so there'll be a follow-up to split that later
* Update Jinja2 to 2.9.6
* Add django-jinja 2.3.1
* move some jingo-minify helpers to olympia.lib to avoid any jingo imports
* unified |f and |fe helpers into one |format_html helper that always escapes everything unless the input is already |safe
* Move helpers, add requirements, fix imports.
* Give backend a proper name
* Port f and fe filters from jingo
* Make start porting reviews jinja helpers module, make first test pass
* Get a handful more tests working, get rid of get_env, more and more jingo calls.
* Disable extension matching, render .txt templates too, fixes more stats tests
* Fix csv_header template to produce some reasonable and csv parsable
* Set trim_blocks to be compatible with jingo
* Unify |f and |fe into one |format that simplifies escaping rules. Update tests.
I think having one unified |format filter is beneficial, especially
given that our current escaping rules were kinda complicated and allowed
for seriously dangerous combinations even with autoescape activated.
Now everything passed as an argument to |format is explicitly escaped
unless marked as |safe, there's no way around that anymore.
* Naive first version using |format instead of |f and |fe, might need more review.
* Fix format for unicode strings
* Rename |format to |format_html
* Move nl2br_xhtml to amo.jinja_helpers, use it in all our templates.
* Move datetime filter to our code temporarily, fix mocks, fix recursive error
* Move csrf call to django_jinja csrf_token tag, fix addons templatetags
* Fix |class_selected usage, fix render_with to get registered
* Remove hack from zadmin, fix zadmin templatetags
* Fix user delete photo test, don't rely on proper context updates that happen in order.
* Fix more nl2br tests
* Fix detail page escape, django escapes differently than jinja
* Remove context from edit_review_reply_form
* Fix date created format filter usage
* Remove templates setting user, fix can_review
* Fix collect_add_widget to not inherit the global context
* Remove happy unicode client
* Cleanup can_review, add tests
* Fix tests, django use different html character for double quote escaping
* Switch to firefox latest to check ui-tests
* Cleanup and fix do_truncate
* Remove jingo, copy jingo-minify helpers that depend on jingo to our lib for now
* Rename nl2br_xhtml -> nl2br
* Jinja 2.9 compat for reviews truncate filter
* Don't use TEMPLATE_DEBUG
* Set default value for q, happens on index page.
* Fix jinja2 formatting with newstyle gettext
It took me days to find this and loads of useless refactoring but
apparently these tasks get immediately upgraded to a chord after we
chain them later with the other tasks (after the for-loop).
And, the only requirement for a chord is: do not ignore the results,
never!
Refs #3960
* Make sure index_stats isn't using call_command
* modernize create_subtasks, add support for countdowns, fix queries loading the world...
* half chunk size for stats reindexing
* Refactor stats reindex logging
* Upgrade elasticsearch to 5.4.1 for our docker config
* Make es test setup easier to inherit for our uitests.
* Restore order, make sure to start es before add-on get's created.
* Remove db requirement from es_test fixture
- Middlewares are integrated to existing ones or moved to amo.middlewares
- Logging is moved to core.logger
- All logging calls are modified to use core.logger
- In addition, get/set_user(), get/set_remote_addr() are moved to core
to circumvent import issues.
* Pull in django-waffle 0.10 and a fixed version of django-piston.
Both provide late-initialization support by not calling get_user_model
on import-time.
* Add alias for addons.buttons.js
We now have something like that:
- /
-- src/
--------- olympia/
-------------- {access,accounts,amo,addons,...}
-------------- wsgi.py
-------------- urls.py
-------------- migrations/
-------------- libs/
...
Which generally feels more django-like again and makes porting
our sys.path hacks to django 1.7 and it's app-refactoring
a lot easier.
* Update setup.py to bring it to a more modern format
* Update gitignore
Christopher Grebs
pyup.io bot
Mathieu Pillard
Mark Striemer
Kris Maglione
Mathieu Agopian