* Serve files in old file-viewer through FileResponse.
Fixes bug 1566954
The old file-viewer will be going away "soon" anyway so we should be
using the same technique as for the new one.
The problem here specifically is that when going through Nginx via
X-Accel-Redirect nginx isn't setting proper CSP headers for the
download. Serving the files ourselves allows us to much easier restrict
CSP in the future even further.
* Explicitly restrict CSP config on download endpoints.
* Set report uri in settings.py too
* Explicitly add frame-ancestors
* Remove outdated test
* Remove all usage of 'six' and as much compat code as I could find.
Cleans up some imports along the way.
Fixes#11728
* Fix typo
* Fix rta related code paths, I actually misread the comment…
* Move ResourceWarning filtering to setup.cfg
* Adds a test to test unicode support for latest python-memcached
versions
* Configures caches key prefix so that tests don't conflict with each other
* Remove a few cache.clear calls with `cache.delete` calls that avoids
deleting all memcached entries while other tests are running
* Remove zadmin.memcached as it's crazy dangerous, such power should
only be wielded by ops-folks who wake up at night when pressed
* Fixes waffle helpers to call .flush() to flush appropriate caches
This also fixes how parameters are forwarded to memcached and fix KEY_PREFIX initialization
We are now making sure parameters from `settings.CACHES` are properly forwarded
to the actual cache backend (now memcached). These errors were not seen
since LocMemCache didn't require any parameters.
The `KEY_PREFIX` setting is set to a uuid instead of the xdist-slaveid to
ensure test functions are properly isolated from each other.
`cache.clear` is not called anywhere unless absolutely necessary to reduce
the side-effects of our test-setup.
Fixes#8602
This ports all amo-validator related code to run via subprocess. This
makes us less blocked by it not running on Python 3, there may still be
some installation / execution related bits to actually run it via python
2 instead of python 3 but that's something for later.
* Cleans up how and where we're setting _BIN settings for all the
binaries we're calling
* Backports `unicodehelper` from validator
* Removes `requeue_uploads` since it's been broken and unused for at
least 3-4 years
* Remove Standalone Add-on Compatibility Checker (/developers/addon/check-compatibility/)
* Remove compat-check related bits from standalone add-on upload
* Remove more bulk compat report stuff that isn't used anymore
* Removed zadmin application_versions.json view, unused
Fixes#9670
"Fixes" #8857
Only the language tools API used the filesystem cache, and that was
because it had a relatively low usage compared to cache-machine, which
meant everything it cached was quickly evicted. Now that cache-machine
is gone, let's use memcached again for this.
Closes#8451 (Doesn't necessary fix it, we don't want a unified logging-level but this PR makes logging much clearer.)
* Don't use verbose mode for tox
* Refactor logging settings
* Fix services logging, correct dev and stage settings, update docs
* Update logging
* Be vigilent about disabling all logging
* Remove MOZLOG_NAME from stage settings
Most of this work landed already in
6a8fc341da and
91605a4c1f so this is just a follow-up to
remove all other traces.
The removed comment is no longer true, Django received all the necessary
fixes for LocMemCache. On the other hand, LoMemCache is no longer used
for a regular local setup because we are using memcached and try to be
as close to production as possible so this won't change anytime
soon.
* Port mozilla/amo-loadtest to FXA authentication and WebExtensions.
This is the first part of many to improve our load-tests and allow them
to be run regularly as part of a regression test suite.
This commit primarily ports the existing tests over to our repository,
updates them to use FxA authentication and uses a WebExtension for the
upload test.
This only implements the baseline for more work to come in #7744 but
it's important to have authentication and the basic infrastructure
working correctly.
It's also still using the legacy add-on detail and listing pages,
that'll obviously change - or be only true for SeaMonkey and Thunderbird
related tests since we still have to support them (very low priority though)
It also adds a first step of a summary report that links to new-relic.
That'll need a bit more tooling and testing but it worked quite well in
first tests.
Refs #7744
Next steps:
* Implement most of the read-only tests from #7744
* Check what needs auth in #7744, implement it properly
* Implement database and cache query logging
* Implement a unified merged test-report that uses the database and
cache query logging and merges it with locust logs (using our new
unique request id)
* Fix makefile, show dummy-usage for now.
* Delete generate-summary script for now
* Pin and cleanup dependencies
* Reverse all urls
* Use passed in account for login
* split things up
* Use gevent for waiting, some cleanups, add first version of Dockerfile, running script, disable developer stuff for now
* Add browsing collections and browsing categories
* Add loads more variation, add support for thunderbird, seamonkey, multiple languages
* Decrease size of docker images for local testing
* Get installation and docker image mostly to work, docs, cleanups
* Allow fxa environment variables be overwritten
* Hit legacy site of every variation
* More explicit theme testing
* Add browsing for app-versions and various rss feeds
* More variation in rss feeds, add featured and search tools
* Add a few more response.success() calls
* Test user profile pages, warn for empty collection pages
* More docs
* Add review browsing
* drop mozilla-logger and all logging1.0 support
* drop (hopefully) unused SYSLOG_CSP setting
* drop 'error' formatter; refactor 'debug' formatter format a little
- Also, use that setting to disable cache-machine by default in tests,
locally and on dev.
- Tweak translations tests to not depend on cache-machine at all
(remove dependency on ModelBase, depend on django models only
+ the translation transformer)
- Allow filtering by type to only show language packs
- Expose a new current_compatible_version if appversion is passed.
- This property will fetch the latest publicly available version
compatible with the appversion passed.
- Replace caching with a filesystem-based implementation to work
around cache eviction issues.
* Update celery, config and dependencies to 4.x
* Remove usage of TaskSet, replace with group(), update `ampq`
* Enable redis on travis for better celery 4.x testing
* Fix lib/celery_tests
* Correct test testing email sending retrying on errors
* Fix celery subtask mocking errors.
* Fix celery broker url env variable
* Integrate watchdog for celery autoreload
* Ensure dependencies are updated for worker and web container.
* Restart all services after the dependency update
* Remove explicit celery loader environment variable.
* Restart all services in web and worker after running 'update_docker'
* Increase sleep time a bit
This reverts commit 541688d5f2.
This caused some breakage on dev. We suspect it's because newrelic
hooks into the celery task launching code but does not support
celery 4.x yet.
* Update celery config to seamlessly upgrade to 4.x
* Update celery initialization for new configs and 4.x
* Update dependencies to celery 4.x
* Fix flake8
* Fix typo
* Remove usage of TaskSet, replace with group(), update `ampq`
* Enable redis on travis for better celery 4.x testing
* Fix more test setting names
* Fix lib/celery_tests
* Correct test testing email sending retrying on errors
* Fix celery subtask mocking errors.
* Fix new line
* Fix celery broker url env variable
* Integrate watchdog for celery autoreload
* Try something weird... ui-tests
* do we need extra hosts?
* more sleep?
* Try logging, try docker run to avoid port collisions
* More debugging
* Cleanup again
* Fix logs?
* Ensure dependencies are updated for worker and web container.
* Remove not functioning log saving for onw
* Restart all services after the dependency update
Loads of clean-ups along the way.
Fixes#1573
* Moves all helpers modules to templatetags.jinja_helpers - there are a few things in there that aren't jinja related so there'll be a follow-up to split that later
* Update Jinja2 to 2.9.6
* Add django-jinja 2.3.1
* move some jingo-minify helpers to olympia.lib to avoid any jingo imports
* unified |f and |fe helpers into one |format_html helper that always escapes everything unless the input is already |safe
* Move helpers, add requirements, fix imports.
* Give backend a proper name
* Port f and fe filters from jingo
* Make start porting reviews jinja helpers module, make first test pass
* Get a handful more tests working, get rid of get_env, more and more jingo calls.
* Disable extension matching, render .txt templates too, fixes more stats tests
* Fix csv_header template to produce some reasonable and csv parsable
* Set trim_blocks to be compatible with jingo
* Unify |f and |fe into one |format that simplifies escaping rules. Update tests.
I think having one unified |format filter is beneficial, especially
given that our current escaping rules were kinda complicated and allowed
for seriously dangerous combinations even with autoescape activated.
Now everything passed as an argument to |format is explicitly escaped
unless marked as |safe, there's no way around that anymore.
* Naive first version using |format instead of |f and |fe, might need more review.
* Fix format for unicode strings
* Rename |format to |format_html
* Move nl2br_xhtml to amo.jinja_helpers, use it in all our templates.
* Move datetime filter to our code temporarily, fix mocks, fix recursive error
* Move csrf call to django_jinja csrf_token tag, fix addons templatetags
* Fix |class_selected usage, fix render_with to get registered
* Remove hack from zadmin, fix zadmin templatetags
* Fix user delete photo test, don't rely on proper context updates that happen in order.
* Fix more nl2br tests
* Fix detail page escape, django escapes differently than jinja
* Remove context from edit_review_reply_form
* Fix date created format filter usage
* Remove templates setting user, fix can_review
* Fix collect_add_widget to not inherit the global context
* Remove happy unicode client
* Cleanup can_review, add tests
* Fix tests, django use different html character for double quote escaping
* Switch to firefox latest to check ui-tests
* Cleanup and fix do_truncate
* Remove jingo, copy jingo-minify helpers that depend on jingo to our lib for now
* Rename nl2br_xhtml -> nl2br
* Jinja 2.9 compat for reviews truncate filter
* Don't use TEMPLATE_DEBUG
* Set default value for q, happens on index page.
* Fix jinja2 formatting with newstyle gettext
* Update compile-mo.sh to understand dbr and dbl locales
* Cleanup omg_new_locale, add support for dbr and dbl debug locales.
* Remove dbg locale
* Add dbr and dbl debug locales
* Add support for dbr, dbl, fix setting for rtl to be LANGUAGES_BIDI as this is what django uses these days
* Remove temporary comment
* Improve debug message
* Disable clean checking for now
* Fix dbg and dbl extracted locales
* Fix dbl generation
* Fix tests
* Revert unnecessary change to zadmin.forms
* Remove cleanness check
* Remove dbr for now, it's horribly broken.
* Various cleanups according to comments
Andrew Williamson
MyeongJun Go
William Durand
Christopher Grebs
Mathieu Pillard
Kumar McMillan
Andreas Wagner
Adrian Utrilla
Mark Striemer