* Run CI/local envs with ES 7 and update client libs
Also drive-by update CI to have the right versions of MySQL/Redis
* Remove deprecated, no longer necessary send_get_body_as parameter
* Fix deprecation warnings
* Update tests
* Unused import
* More deprecation fixes
* Import addons-nginx into addons-server
* Serve xpis through X-Accel-Redirect instead of redirecting to dedicated domain
The so called "CDN URLs" for files are less useful now that even AMO main domain
is behind a CDN, and we want to move away from it because internally it exposes
paths on EFS that nginx is serving directly, making directory structure changes
difficult.
This commit focuses on the download_file/download_latest endpoints
addons-server exposes, and leaves the old dedicated CDN domain intact for the
moment - it is still used by the update service but will go away in a future
commit.
* Few fixes and test updates
* filename is an optional argument
* Test fixes
* Add assertions about caching
* Fix latest url
* Rename type_ to download_type
* reformat url definition
* Upgrade to Elasticsearch 6.x
Because ES 6.x removed mapping types, move update_count and download_count in
separate ES indexes. This requires a full reindex, including stats, but we need
to create an entirely new cluster anyway.
ES 5.x compatibility is kept for now.
This updates various things…
* make use o `--user` argument of pip and `$PYTHONUSERBASE` to install python libraries in the correct places where the `olympia` user has permissions
* Add a new `rootshell` command
Fixes#12366
* Correct if statement
* Use separate tag for easier testing
* Try running directly with user olympia
* Small cleanup
* Add docs
* Fix comment
* Add 'rootshell' command
* Build circleci for local dev
* Fix pip execution by using 'python -m' to fetch the correct pip
* Pick up upstream user/group for builds and implement PYTHONUSERBASE and PIP_USER
* Correctly pass USER_ID and GROUP_ID to Dockerfile when building
* Small cleanups
* Build our test branch
* Fall back to manual uploads for testing
* Cleanup
* Revert back to :latest tag
* Move CI envvar definition to selenium override config.
This should fix autoreload.
* Actually use to be more explicit
* Disable integration tests
* Try out different approach
* Add docs, move back the variable definition to docker-compose.yml
* Update comment
* Sign add-ons with recommendation signer based on DiscoveryItem.
This also prepares for the possibility that we may have to use different
credentials for the recommendations signer. In case we don't, ops only
has to set the creds twice which seems fair.
Fixes#11062
* Unify travis and docker-compose configs, test separate signer hawk creds
* Add end-to-end test for reviewer tools signing
* Enable autograph for reviewers-and-zadmin
* Remove amo-validator, remove more support for uploading legacy-addons.
This is dropping most of our compatibility checks for legacy add-ons and
completely blocks them from being uploaded.
We workaround this block for mozilla-signed add-ons by issuing the same
error, just as a warning instead of an actual error.
This affected *loads* of our tests so the PR contains more test-changes
then I'd like.
In addition to that this PR adds file extension validation for uploads,
we only did that in our JavaScript and never in our code. So from now
on, we have a hard limit to only allow whatever is in
`VALID_EXTENSIONS`.
Further...
* drop amo-validator and related dependencies
* remove `dump_apps` management command
* removed unused fixture files from the repo
Fixes#10502
Future improvement idea for our test-suite: Unify add-on file fixtures
under one single folder to avoid any duplication.
There is most certainly some more improvements that could be made to
code and test structure but I went for the least changes, it's way too
huge already :-/
* Fix syntax error in JavaScript, thus ui tests
* Switch back to passing 'channel' around
* Move valid addon file extensions to constants/base
* Fix validate() call
* Fix more tests
* Update src/olympia/devhub/tests/test_views_validation.py
Co-Authored-By: EnTeQuAk <cg@webshox.org>
* Remove old comment
* Issue description specific only to Thunderbird and Seamonkey if targeting thunderbird or seamonkey
This doesn't run services yet, and is just useful to run tests. Because
it shares the same image as the regular web container, an initial
make update_deps is necessary to get it working. After that, running
pytest as normal should function normally and be using Python 3.
Unfortunately, we'll have to duplicate the environment config but I added
a comment so that should be fine. It's very rarely being used anyway.
Fixes#9288
* Changed fxa account generation to allow assigning an fxa ID to a created user.
* Moved pytest execution to makefile to access variables.
* Removed second login tests as the devhub tests also log in.
* Default to 'and' operator for match queries. Remove slug search, prioritize exact matches more.
References many "component: search" issues. What I tested with a
database of all public add-ons:
Example searches:
tab center redux - should find "Tab Center Redux" while "Tab Mix Plus" is probably second and "Redux DevTools" 4th or so
Open Image in New Tab -> should find "Open Image in New Tab" while "Open Bookmarks in New Tab" should be 2nd or 3rd
CoinHive -> Finds "Coinhive Blocker", "CoinBlock" (prefix search) and "NoMiners" (description)
Privacy -> Finds "Privacy Badger", "Privacy Pass", "Privacy Settings", "Google Privacy" (probably 4th or so) and "Blur" (summary + description + many users). Scores "Ghostery" on the first page but ranks it in the middle
firebu -> Finds "Firebug", "Firebug Autocompleter", "Firefinder for Firebug"
fireb -> Scores "Fire Drag" first, puts "Firebug" approximately 3rd or so
Menu Wizzard -> Finds "Menu Wizard" (fuzzy, typo) first, then "Add-ons Manager Context Menu" apparently because it matches good in the title and has many users
Frame Demolition -> Finds "Frame Demolition"
Demolition -> Finds only "Frame Demolition", same for "Demolation" (typo)
reStyle -> Finds "reStyle" and scores a few add-ons that match on "restore" next since the term is similar
MegaUpload DownloadHelper -> finds "MegaUpload DownloadHelper" first, scores "Video DownloadHelper" and "RadpidShare DownloadHelper" next. Doesn't find "Popup Blocker" anymore as currently happening on -prod
MegaUpload -> only finds "MegaUpload DownloadHelper" and nothing else
No Flash -> Scores "No Flash" first, then depending on users "Download Flash and Video", "YouTube Flash Video Player" and "YouTube Flash Player" (not necessarily in that order)
Disable Hello, Pocket & Reader+ -> finds "Disable Hello, Pocket & Reader+" first (yeay!), then scores "Reader", "Disable WebRTC" and "In My Pocket" next similarly to what's happening on -prod currently
Not working yet:
privacybadger -> "Privacy Badger" -> will probably need some kind of ngram filtering and analyzing (#591)
eyes -> 'decentraleyes' -> Not sure this should actually work, will probably need some more analyzing too (#591)
Not sure if it's specifically only because of these changes but #3248 is fixed.
This potentially fixes#7244, #6891, #6837, #6417, mozilla/addons#359.
Not sure if this fixes #mozilla/addons#567 but the results look much more promising and the amount of results doesn't explode here. I only have 2.8k add-ons for testing though so I'm not too sure.
And might be relevant to #6137.
This is a big step towards #2661, I doubt we can call this fixed though.
* Fix name tests
* Speed up ES tests, make scoring results more predictable by using only one shard and one replica
* Add tests
* Only test on new apiv3 based search.
* Fixup 'get_results'
* Remove debug print
* Isort imports
* Fix test settings, I'm blind.
* Adapt number of shards to what we define in settings_test
* Test search for grapple
* Remove property filtering in legacy api search, fix tests, fix flake8
* Add comment explaining shard config
* Fix tests again, add default platform, fix total counts again.
* Try to do some fixture cleanup, let's see...
* More test refactoring, make use of dfs-query-then-fetch during tests, allow us to test this later too via a waffle flag.
* Don't use waffle flag for legacy search, only for apiv3 searches.
* Fix unused import, fix usage of 'params'
* Use query-then-fetch in a regular addons-view tool, fix a few more tests to be able to handle the waffle-flag
* Create the dfs query then fetch flag only in ESTestCase, delete it properly. More serializer fixes
* precache the waffle flag for autocomplete tests too
* Minor cleanups
* Add docs, fix code style
* Add a todo
* Fix codestyle
Mathieu Pillard
Andreas Wagner
William Durand
Andrew Williamson
mirefly
dbxnr
Christopher Grebs
Benjamin Forehand Jr
Mathieu Pillard
Greg Guthe