* Rework email blocking to allow blocking whole domains too.
This works very basically also for subdomains.
This doesn't support wildcards.
Fixes#11442
* Use fnmatch to match on glob-style patterns.
Fixes#11484
* Fix tests and flake8
* Rename domain_pattern to email_pattern as that fits better.
* just change the email column
* Fix migration number
* fix admin
This forces restricted users (from their email or current IP) to go to the
developer agreement page when they want to use the submission flow. When
the agreement page is submitted, they'll be shown a message depending on
the reason behind the restriction.
A new API permission is also added that completely prevents the signing API
from allowing requests from such restricted users, with the corresponding
message returned as well.
* First part of adding user restriction model and admin.
This adds a `UserRestriction` model that allows admins to add
restrictions based on ip-address, cidr network or email.
The admin is currently very basic and will most certainly be improved
over time.
Fixes#11440 (already contains CIDR support)
Refs #11442 (missing email domain support, going to push that in a
follow-up)
* Split up user restriction checkers. Add actual allow_* implementation and tests
* Improve tests, fix /32 blocking
* Merge IPAddressUserRestriction and IPNetworkUserRestriction
We should never hit that code in real life for now, as you need Users:Edit
permission to see the list anyway, but this was wrong. (There is an
additional permission check in the method itself so there wasn't any
security risk)
The individual collection pages no longer exist in the new frontend,
so we need this link to point to something that does and this is the
one that makes the most sense.
* Adds a test to test unicode support for latest python-memcached
versions
* Configures caches key prefix so that tests don't conflict with each other
* Remove a few cache.clear calls with `cache.delete` calls that avoids
deleting all memcached entries while other tests are running
* Remove zadmin.memcached as it's crazy dangerous, such power should
only be wielded by ops-folks who wake up at night when pressed
* Fixes waffle helpers to call .flush() to flush appropriate caches
This also fixes how parameters are forwarded to memcached and fix KEY_PREFIX initialization
We are now making sure parameters from `settings.CACHES` are properly forwarded
to the actual cache backend (now memcached). These errors were not seen
since LocMemCache didn't require any parameters.
The `KEY_PREFIX` setting is set to a uuid instead of the xdist-slaveid to
ensure test functions are properly isolated from each other.
`cache.clear` is not called anywhere unless absolutely necessary to reduce
the side-effects of our test-setup.
Fixes#8602
This should fix most of our current Django 2.x and DRF deprecation warnings that we see during test runs.
* Uses `remote_field.model` in amo/utils instead of `remote_field.to`
* Uses `is_anonymous` and `is_authenticated` as properties instead of functions
* Replaces drf `@detail_route` with `@action(detail=True)`
Fixes#9432
* Changed fxa account generation to allow assigning an fxa ID to a created user.
* Moved pytest execution to makefile to access variables.
* Removed second login tests as the devhub tests also log in.
Note that the behaviour depends on which model is affected - some are kept,
others are deleted because of UserProfile.delete_or_disable_related_content().
- Refactor auth register and login to *both* send `user_logged_in` signal
- Use that signal to set `last_login_ip` (django uses it to set `last_login`)
- Also send that signal when authenticating through JWT tokens
- Deprecate the `*_attempt` fields that no longer make sense with FxA auth
Fix#7333
Port code and tests to Django 1.11.
* Update requirements to Django 1.11.14, remove session_csrf
* Use --run-syncdb for migration calls
* Remove unused setup-docker script
* Port code and tests to Django 1.11.
* Remove django-session-csrf everywhere, it got replaced by django's own
session csrf protection
* Make sure we are calling `client.get()` with `follow=True` where necessary
* Replace django.core.urlresolvers import with django.urls
* Port custom `IconWidgetRenderer` to Django 1.11 as `IconTypeSelect`
* Properly use `_queryset_class` for custom manager classes
* Make use of `base_manager_name` on our models `Meta` instead of `use_for_related_objects`
on the manager where appropriate. This is required to make translations, soft-delete
and more work. See comment on Addons.Meta for more details.
* Various fixes along the way to make tests work
* Port form errors for locale editing forms to Django 1.11 and make use of
form.add_error instead of our own custom process
* Mark TestDetailPage.test_304 as xfail for now, somehow Etags seem to be broken
for that view during tests but work fine in manual tests
* Fix amo.celery, we cannot have any imports at the module-level that may
import the ORM
* Fix global_settings context processor for 500-server error cases
* Fix `changed_data` handling in `AMOModelForm`
* Remove `queryset_transform` and implement it directly in amo/models
* Fix pagination, Django changed how `num_pages` work and how `count` can be overwritten
* Add a new test that tests database transforms as `BaseQuerysetTestCase`
* Make sure `legacy_redirect` only appends `?` if there are actual GET parameters
* Re-implement `LicenseRadioChoiceInput` as `LicenseRadioSelect` which is now Django 1.11 compatible
* Remove our custom `form_kwargs` uplift from Django 1.9
* Correct how we're using `Category.objects.create` and only pass `db_name` instead of `name`, previously
this didn't error but now it (luckily!) does
* Update `FileSelectWidget` to correctly work with Django 1.11, use custom rendering
* Fix decorator ordering for `files.views:browse`
* Update database settings to correctly use `TEST` object
* Fix `select_related` call in `RatingViewSet`
* Remove custom `DateTimeCast` implementation
* Fix `addon_factory` calls in `reviewers/tests/test_views.py` to correctly set
`is_webextension`
* Port our `TranslationDescriptor` and `TranslatedField` to Django 1.11. Also, make sure form error handling works
* Add a new test that checks that `default=None` is a valid setting for translation fields
* Add a new test that checks that translations from related fields are correctly fetched
* Remove possible ambiguity from translations `TestWidget` tests
* Make sure that the correct `MEDIA_URL` is set in `TestDownloadsUnlistedVersions` to fix
external cdn tests
* Remove custom happyforms code. Django cleans text fields now by default
* Move translations app to the top of INSTALLED_APPS, add a lengthy comment
* Pass additional arguments to contribute_to_class, fix smaller nits
* Set MEDIA_URL and STATIC_URL to an external value by default. That's how they're configured in dev, stage and prod anyway.
Please note that we didn't replace all our custom form rendering with proper templated widgets yet. This will
happening in some follow-up work.
Fixes#5271, Fixes#1569
This in turns means adding the notion of banned users and preventing
them from logging on. They are just deleted users for which we keep
the email and fxa_id.
Part of our upgrade to Django 1.11, this is backporting functionallity
we'd like to use after the upgrade already to our current code so that
we can find potentially drawbags early.
This also reduces our reliance on cache-machine which helps removing it.
This also few smaller optimizations along the way:
* denied-name blocklists were cached language-dependend although they're not, this now increases the chance of cache-hits instead of misses
* the django cache.get_or_set implementation fixes potential
race-conditions between multiple requests setting data
Fixes#8529
* Remove template fragment caching.
* Removes template fragment-caching completely
* Adds a few performance optimizations, backported from #8435
* Removes the globally available `ADMIN_MESSAGE` and explicitly fetches
it in our base template. That way we don't call zadmin.get_config on
every template-render needlessly
* Improved developer-status querying, merged the caching from is_developer, is_addon_developer, and is_artist
Fixes#8460
* Actually remove amo.ext module.
* Revert not fully-completed addons_with_authors rework
* Fix comment
2 things were wrong here:
- We didn't use basket data if the state was 'unsubscribed'
- We didn't check against the basket_id so the if was never True
The form had both mistakes, the API only the second one.
Swallowing basket errors or letting it do async calls is
dangerous for us, because the user is not aware if something
went wrong. Better for them to get a 500 page if things are
not working properly.
* Fix basket integration for unknown users.
Fixes#8281
This also removes the orphaned sync_user_with_basket task. We do that
synchronisation on-demand and during the request when a user calls
it's user profile page.
* Return empty list and don't return empty tuple when sync on unsubscribe didn't work
* Clear more personal data on user deletion.
Fixes#8223
* Fix failing test thanks to modified fixture data
* Also delete photos on user deletion.
Fixes#8222
* Add a command that re-calls user.delete() on all deleted users.
* No need anymore to name the task explicitly
* Fix flake8, correctly delete user-images in case of hard delete
* Fix docstring
* Implement basket <-> amo newsletter sync.
* Adds basket-client dependency
* Refactors users.notifications to allow for remotely-managed
notification types
* Implement helpers to fetch, subscribe and unsubscribe from newsletters
* Update user profile editing form to fetch and save to basket
* Update accounts/notifications endpoint to fetch and save to basket
* Implement a sync_basket command that fetches basket tokens
asynchronously for all users
All this is behind a waffle flag.
Fixes#7808
* Switch to 100% synchronous basket requests again
* Fix accounts view test to use the conftest configured token
* Fix typos and comments
* Put basket integration behind waffle flag
* Fix typo
* Fix tests
* Remove sync_basket command
* Fix flake8
* Implement cache request tracking, add zadmin config tests.
Implements CacheStatTracker as a backend for tests to allow us to use
`assert_cache_requests` in tests.
* implement a `cached` helper and removes caching.{cached,cached_with} instances.
* change from caching backends back to django cache backends
* Move cache_helpers -> cache_nuggets
* Fix CacheStatTracker to not remove OPTIONS values
Move "staff" admin into regular django admin & add Admin:Curation permission
In addition, to allow this to happen, refactor the way we deal with django
permissions as needed by the django admin, by adding a mapping to our own
permissions system. At the moment the only mapping really needed is for
Admin:Curation, which allows full access to ReplacementAddon objects in
the admin.
Because the django admin can now properly show you and give you access
to only what you have the permissions for, the "staff" admin is no longer
necessary and removed in this commit.
* Add isort to our codebase.
This is the first draft, please feel free to discuss this but I'm
generally open to change that style later on.
I just want to start making more and more of our code-style a rule
instead of something we have to take care of during code reviews.
* Ignore safe_xml file
* Don't change safe_lxml_etree
* Fix import unifification going bonkers, I don't blame isort though, this was creepy.
* Fix signin.test_views
* More flake8
* Use style number 4, hanging grid
* django group only contains django now, move above 3rd party group
* Remove pytest-isort for now to avoid isort being imported during tests.
isort has a super bad module that sets the defaultsystemencoding to
utf-8 and breaks our tests that ensure that our settings are *never*
unicode strings. If the default system encoding is set to utf-8 these
tests just work since the bytestring will be implicitly encoded to utf-8
instead of ascii which is the python2 default.
https://github.com/timothycrosley/isort/blob/develop/isort/pie_slice.py#L145
* Rename and re-organize reviewer groups
- Rename Add-on Reviewers as Reviewers: Legacy, and only give them
Addons:Review permission.
- Create Unlisted Add-on Reviewers as Reviewers: Unlisted.
- Rename Persona Reviewers as Reviewers: Themes.
- Remove the Senior Personas Reviewers group and SeniorPersonasTools:View
permission (all Personas reviewer tools are available to all personas
reviewers).
- Remove the Senior Add-on Reviewers group.
- Create a Reviewers: Content group with the permission Addons:ContentReview.
- Create a Reviewers: Add-ons group with the permission Addons:PostReview.
- Create a Reviewers: Moderators group with the permission Ratings:Moderate.
That permission replaces the old rules for moderating user reviews.
- Remove the obsolete concept of 'Limited Reviewers'
* Update celery, config and dependencies to 4.x
* Remove usage of TaskSet, replace with group(), update `ampq`
* Enable redis on travis for better celery 4.x testing
* Fix lib/celery_tests
* Correct test testing email sending retrying on errors
* Fix celery subtask mocking errors.
* Fix celery broker url env variable
* Integrate watchdog for celery autoreload
* Ensure dependencies are updated for worker and web container.
* Restart all services after the dependency update
* Remove explicit celery loader environment variable.
* Restart all services in web and worker after running 'update_docker'
* Increase sleep time a bit
This reverts commit 541688d5f2.
This caused some breakage on dev. We suspect it's because newrelic
hooks into the celery task launching code but does not support
celery 4.x yet.
* Update celery config to seamlessly upgrade to 4.x
* Update celery initialization for new configs and 4.x
* Update dependencies to celery 4.x
* Fix flake8
* Fix typo
* Remove usage of TaskSet, replace with group(), update `ampq`
* Enable redis on travis for better celery 4.x testing
* Fix more test setting names
* Fix lib/celery_tests
* Correct test testing email sending retrying on errors
* Fix celery subtask mocking errors.
* Fix new line
* Fix celery broker url env variable
* Integrate watchdog for celery autoreload
* Try something weird... ui-tests
* do we need extra hosts?
* more sleep?
* Try logging, try docker run to avoid port collisions
* More debugging
* Cleanup again
* Fix logs?
* Ensure dependencies are updated for worker and web container.
* Remove not functioning log saving for onw
* Restart all services after the dependency update
* Replace SortedDict with OrderedDict
* Replace get_field_by_name() by get_field()
* Consider warnings as errors to catch DeprecationWarnings
* Upgrade pytest django, since it generates a lot of deprecation warnings
* Ignore DeprecationWarning from django-csp
* Replace old option parsing syntax in management commands
* Replace request.REQUEST with request.POST
* Replace custom sql to create indexes by model options
* Replace get_all_field_names() with get_fields()
* don't keep this
* Try removing the create-db, we're having errors at teardown
* Fix commands args parsing
* Re-consider DeprecationWarnings as errors
* Remove usage of Context()
* Fix DeprecationWarning caused by paypal PaypalError self.message
* Fix linter test using exception.message
* Fix mock to prevent passing a callable to a queryset filter
* Remove reference to non-existent fixture
* django.forms.util module has been renamed. Use django.forms.utils instead
* (WIP) Support zh-hant & zh-hans instead of zh-TW and zh-CN, respectively.
* Don't pass callable to queryset
* ignore jwt deprecation warnings
* Don't use TestCase.urls anymore
* test against exception message directly
* Simplify nested context managers
* Restore this usage of Context(), it's still valid
* Restore setup.cfg
* Revert "(WIP) Support zh-hant & zh-hans instead of zh-TW and zh-CN, respectively."
This reverts commit 7d82b59850.
* fix flake8
Loads of clean-ups along the way.
Fixes#1573
* Moves all helpers modules to templatetags.jinja_helpers - there are a few things in there that aren't jinja related so there'll be a follow-up to split that later
* Update Jinja2 to 2.9.6
* Add django-jinja 2.3.1
* move some jingo-minify helpers to olympia.lib to avoid any jingo imports
* unified |f and |fe helpers into one |format_html helper that always escapes everything unless the input is already |safe
* Move helpers, add requirements, fix imports.
* Give backend a proper name
* Port f and fe filters from jingo
* Make start porting reviews jinja helpers module, make first test pass
* Get a handful more tests working, get rid of get_env, more and more jingo calls.
* Disable extension matching, render .txt templates too, fixes more stats tests
* Fix csv_header template to produce some reasonable and csv parsable
* Set trim_blocks to be compatible with jingo
* Unify |f and |fe into one |format that simplifies escaping rules. Update tests.
I think having one unified |format filter is beneficial, especially
given that our current escaping rules were kinda complicated and allowed
for seriously dangerous combinations even with autoescape activated.
Now everything passed as an argument to |format is explicitly escaped
unless marked as |safe, there's no way around that anymore.
* Naive first version using |format instead of |f and |fe, might need more review.
* Fix format for unicode strings
* Rename |format to |format_html
* Move nl2br_xhtml to amo.jinja_helpers, use it in all our templates.
* Move datetime filter to our code temporarily, fix mocks, fix recursive error
* Move csrf call to django_jinja csrf_token tag, fix addons templatetags
* Fix |class_selected usage, fix render_with to get registered
* Remove hack from zadmin, fix zadmin templatetags
* Fix user delete photo test, don't rely on proper context updates that happen in order.
* Fix more nl2br tests
* Fix detail page escape, django escapes differently than jinja
* Remove context from edit_review_reply_form
* Fix date created format filter usage
* Remove templates setting user, fix can_review
* Fix collect_add_widget to not inherit the global context
* Remove happy unicode client
* Cleanup can_review, add tests
* Fix tests, django use different html character for double quote escaping
* Switch to firefox latest to check ui-tests
* Cleanup and fix do_truncate
* Remove jingo, copy jingo-minify helpers that depend on jingo to our lib for now
* Rename nl2br_xhtml -> nl2br
* Jinja 2.9 compat for reviews truncate filter
* Don't use TEMPLATE_DEBUG
* Set default value for q, happens on index page.
* Fix jinja2 formatting with newstyle gettext
Update developer agreement / submission process for post-review
Every developer need to read the new agreement page again, which
now contains review rules & policies as well.
* Replace our implementation of @cached_property by django's
* Fix groups_list cached property deletion, add a test
* Remove useless / broken deletions of cached_properties
In those cases, deleting the property is useless either because
it did not previously exist or because we're not doing anything
with the instance afterwards.
* Move invalidate_groups_list to a method on GroupUser
That property was not super useful nor used much anyway. It's important to
not use @cached_property for querysets because it can break lazy evaluation
and cause surprises...
Fix#5480
Fixes#1574
* First part of refactoring... incomplete
* Some more ugettext/ugettext lazy work, still wip
* Last pass on ugettext/ugettext_lazy unification
* Fix last occourences in bandwagon.forms
* More flake8 fixes
* Fix failing unittests
* Lots smaller fixes and replacements from review comments
This mimics how django creates session tokens, except a different salt is used.
When a user auth_id is changed, its sessions and tokens are no longer considered
valid.
Note: JWT created by external clients using their API keys are still valid, but
that's a different auth mechanism only used by a handful of APIs, like add-on
signing.
- Middlewares are integrated to existing ones or moved to amo.middlewares
- Logging is moved to core.logger
- All logging calls are modified to use core.logger
- In addition, get/set_user(), get/set_remote_addr() are moved to core
to circumvent import issues.
addon_factory() randomly creates categories and assign addons it
creates to them. Tests were failing because they were manually
creating the addoncategory instance, which in some cases already
existed.
Filtering should be done either with the status from now (pure
unlisted add-ons will have STATUS_NULL) or by checking if each
add-on has listed versions or not (with .has_listed_versions()),
or by checking if current_version exists, depending on the
use-case.
Fix#4028
This has the side-effect of breaking download_latest view for
unlisted addons, since it relied on current_version. From this
point forward this view will always return the latest *listed*
version (it's necessary to make a decision about its behaviour
anyway since we're mixing unlisted/listed versions, and this
preserves backwards-compatibility for listed behaviour).
Also add process_addons --task=update_current_version_for_unlisted
command to reset the field on unlisted addons we have right now.
* Get rid of addons.query - this will need some benchmarking
but it was super ugly so I removed it
* Make sure email fields are using 75 char maxlength (original default)
* Add replacement for shortcuts.render
* various stuff along the way...
* Fix zadmin render helper
* Less django.shortcuts:render, more our own for compat reasons.
* Fix dynamic endpoints compat
* Fix render support for blocklist
* Save translated fields before we actually save the instance.
This avoids getting caught by django's sanity check that avoids
an instance being saved with unsaved related fields set.
* Fix fix_let_scope_bustage management command
* Fix another horribly wrong select_related statement
* use django UUIDField where possible
* Fix password reset email rendering
* Fix slugify to work with translation objects
* Fix stats csv test with updated cache-control header
* Accept both, verbose and hex version of rss keys
* Add migration to normalize devhub rss keys
* Support both formats with and without microseconds for handle_date
* Copy over render_to_string from jingo and fix it.
* Unify usage of smart/force text/bytes, also to make sure 'Translation' objects are passed through properly.
* Make translation model json renderable
* Unify JSONEncoder usage
* Remove refinements helper is no where used
* Remove unused SeparatedValuesField
* Remove some more unused and untested helper code.
* logging now has a NullHandler.
* Remove unused lib.licenses
* Use bleach.clean instead of our copy of django's remove_tags
* Updated all code that used chardet to not use it anymore
* Port fudge to mock or even remove it
* pymysql is unused
* importlib-no-failures was only a requirement for python 2.6
Mathieu Pillard
Andreas Wagner
Mathieu Pillard
Christopher Grebs
Andrew Williamson
Shivam Singhal
pyup.io bot
Yash Sharma
Benjamin Forehand Jr
lavish205
rafspiny
Sanyam Khurana
Vishal Sharma
Hemanth Kumar Veeranki
Mark Striemer
atiqueahmedziad
Mark Striemer
Andy McKay
ka7
Mathieu Pillard