* Bump wagtail-localize-smartling to 0.4.0
* For VPN CTA snippet, make the string name include the locale, to improve Admin UX
* Stop Django from downcasing the VPN Snippet model name, in order to improve Admin UX
* Fix bug where falling back to a non-CMS page in prefer_cms caused a 302 to en-US
* Stop wagtail-localize from publishing a non-default locale page when adding to the page tree
---------
Co-authored-by: Steve Jalim <stevejalim@mozilla.com>
* Add support for animated GIF processing in Wagtail with Wand
* Stop sending the BedrockImage object instance into the task queue
Instead, just send its ID and then pluck the model from the DB as part of the task
This will mean that CMS notifications will go via the task queue.
The default queue of, well, "default" will be used, which is
one of the queues already being processed by our worker pods: https://github.com/mozilla/bedrock/blob/main/bin/run-worker.sh#L8
At the moment, the Web deployment does not have Redis available to it
so will not have a task queue enabled, and so in turn the
emails that web pods send (eg press requests) will have
no change to their sending pattern
* Add wagtail-localize-smartling to the project, for Smartling L10N support
* Add docs for wagtail-localize-smartling
* Minor reformatting
* Fix duplicated env var reference 🤦
* Add a management command that wraps the sync_smartling command so we can monitor it with a DMS
Wagtail 6 gives us better list-based management of some DB-backed data,
which will be very handy for managing translation jobs with
wagtail-localize-smartling, so upgrading is now essential.
Note that 6.2 LTS is due on 1 August 2024 anyway.
This opens the door to connection pooling without PgBouncer once we're on Django 5.1, but should also be
faster than v2 anyway, so worth trying now to deal with our DB saturation issues.
Small dep bumps along the way.
* Add mozilla-django-oidc to the project dependencies
* Add SSO support to Bedrock for accessing Wagtail and Django admins
* Plumbs in mozilla-django-oidc
* Add custom login pages for Wagtail and Django admins that show an SSO button instead of form fields
* Retain support for username + password login (for local development)
* Tests
* Add custom CSRF page to help explain SSO-related session loss, if it occurs
Because a renewed/cycled OIDC/SSO session can zap a CSRF token and block
a user from submitting a CMS edit, we need to provide a bit more information
about what's happened. This changeset adds that, via a new template and a tiny view
to serve it, plugged in as Django's default CSRF view
Logged out users (who are very unlikely to see this anyway) get a simple
version of the message, while logged in users get more detail/context.
* Bump SSO lease time to 18 hours - trying to balance awkward signouts with wanting re-checks
* Update test.env so that Wagtail and Django admins are available by default when urlconf is generated. Oddly the reload trick didn't work here
* Update bedrock/base/templates/403_csrf.html
Co-authored-by: Alex Gibson <alexgibson@users.noreply.github.com>
* Make translation tagging consistent on new login templates
* Move new CSRF view to use a CSS bundle, not inline CSS
* Remove old, redundant CSRF view
It looks like this was no longer in use. It wasn't specified as settings.CSRF_FAILURE_VIEW so wouldn't have been used/found by Django I believe
* Drop translation markup from login templates to simplify
* Don't count the test 404 and 500 views as nonlocaled, because we do localize them
* Update bedrock/admin/templates/wagtailadmin/login.html
* Tweak wording re SSO for login pages
---------
Co-authored-by: Alex Gibson <alexgibson@users.noreply.github.com>
Production deps
Resolves#14634 - Bump lxml from 5.2.1 to 5.2.2
Resolves#14630 - Bump newrelic from 9.9.0 to 9.10.0
Resolves#14624 - Bump requests from 2.32.2 to 2.32.3
Resolves#14623 - Bump dj-database-url from 2.1.0 to 2.2.0
Resolves#14622 - Bump wagtail-localize from 1.8 to 1.9
Development deps
Resolves#14633 - Bump ruff from 0.4.2 to 0.4.7
Resolves#14632 - Bump wagtail-factories from 4.1.0 to 4.2.1
Resolves#14631 - Bump pipdeptree from 2.19.1 to 2.21.0
Resolves#14525 - Bump compare-locales from 9.0.2 to 9.0.3
Resolves#14519 - Bump translate-toolkit from 3.12.2 to 3.13.0
* Add django-rq to the project, with initial configuration for two redis-backed queues if redis is available
* Add Django-rq to project, including link from (now necessary) Django admin's userlinks
* Update Docker support to run a local worker node for RQ
* Add Sentry integration for Redis and for RQ
* Add a make command to run the worker queue
Temporarily loads env vars from .env, looking for REDIS_URL
* Make a new setting that makes it unambiguous whether a task queue is available
* Add helper that will enqueue a task or immediately execute it, depending on whether the task queue is available
Note that if it finds a queue is available it will put a job in there, regardless of whether the queue is being processed by a worker
* Switch image-rendition pre-generation to use the task queue, if available
* Add initial, basic, docs about the task queue
* Improve logging around task queue usage
* Fix formatting of Makefile
Thanks @janbrasna!
* Typo fixes in documentation
Thanks @janbrasna!
* Add Wagtail 5.2 LTS as a dependency
* Drop in wagtail-localize, ready for i18n/l10n
Also fix up some test regressions as a result of this change, including
changing how Careers renders a custom 404-page message
* Support detection of CMS-based pages so that we don't falsely redirect away from pages managed by wagtail-localize
* Ensure that pages fall back (in their enriety) if we don't have a page with the matching lang code in the CMS
* Back out hack to 404.html for careers-related 404 -- unnecessary following i18n/routing refactor
* Back out unnecessary follow=True on the 404 test
* Make database config dynamic, but default to current local sqlite DB
For most setups, without a DATABASE_URL set, nothing changes
* Add psycopg3 dependency, for postgres support
* Avoid hitting the productdetails table during bootup
Because this makes it impossible to boot Bedrock with an empty database
* Add postgres DB to docker-compose, initially for local-dev use only
* Support bootstrapping an empty postgres database with bedrock data
* Update models in security and contentful models' schemae to handle invalid
data that sqlite tolerated, but postgres does not
* Add a shell script that will bootstrap an empty postgres database with
data either from the main external sources, or from the sqlite DB that
bedrock downloads
* Add db-to-sqlite lib to project
Downgrade psycopg(3) to psycopg2 due to hard dependencies of db-to-sqlite
* Rename postgres-bootstrapping commmand to kebab-case for consistency
* Add script to export postgres DB to sqlite, redacting auth_user table along the way
* Expand DB export script to redact or skip certain Wagtail-related tables/columns
* Switch to using loaddata/dumpdata instead of db-to-sqlite, because we were having column-type drift that was breaking Wagtail
* Tune the dumpdata export a little
Based on https://github.com/wagtail/bakerydemo?tab=readme-ov-file#preparing-this-archive-for-distribution
* Rename postgres-filling command to be less ambiguous
* Include ProductRelease in dumped data, because now we can after fixing the NotesField in an earlier commit
* DRY up the success/exit code
* Revert unnecessary lazy-making of product_details
* make export-to-sqlite script more robust and clearer
* Add simple guard rails to ensure DATABASE_URL points to a postgres DB when these scripts are run
* Add test coverage for data-scrubber management command
(Originally written by Copilot but heavily refactored and made appropriate)
* Allow disabling Markus when DEBUG=True (default: keep it enabled)
* Add Wagtail 5.2 LTS as a dependency
* Hook in Wagtail Admin, while also keeping jinja-template rendering happy
* Support user-uploaded images into the CMS, both with local and cloud storage options
Of course, the cloud storage option is the only one we'll use when deployed.
Note that this also disables CSP for any routes inside the Wagtail admin
(but not pages rendered by Wagtail)
* Drop in wagtail-localize, ready for i18n/l10n
Also fix up some test regressions as a result of this change, including
changing how Careers renders a custom 404-page message
* Lay in intial 'cms' app as home for our Wagtail-related modelling.
* Support detection of CMS-based pages so that we don't falsely redirect away from pages managed by wagtail-localize
* Wrap simple CMS test page in a Protocol-based base template to confirm we load the right strings and get the lang picker
* Ensure that pages fall back (in their enriety) if we don't have a page with the matching lang code in the CMS
* Exclude the cms-admin path from the sitemap - doesn't need to be discoverable
* Remove now-redundant print statement from a data migration
* Add tests to show locales from page models are used as translations for pages
* Add tests to show that pages with restrictions (e.g. passwords) should not be cached downstream
* Add tests for StructuralPage, confirming it redirects to its parent if accessed directly
* Expand test coverage to include serve_preview for our base page model
* Fixup accidentally-commented-out app in INSTALLED_APPS
Co-authored-by: Ryan Johnson <escattone@gmail.com>
* Improve comment related to local GCS credentials, clarifying who will/won't need them and why
* Disable use of Gravatar URLs for Wagtail users
Low benefit to us, because they need CSP tweaks and also would need scrubbing before the DB export.
* Fix labelling of Wagtail's homepage
* We don't want wagtailforms enabled in our setup: public bedrock is read only
* Back out hack to 404.html for careers-related 404 -- unnecessary following i18n/routing refactor
* Back out unnecessary follow=True on the 404 test
* Ensure that demos run migrations on their local DB, so that any new Wagtail tables get made ASAP, preventing 500s
* Add a fake spec for WAGTAIL_CONTENT_LANGUAGES in Pocket mode, so that it'll boot.
We won't be using wagtail for Pocket mode
* Rename MEDIA_URL from user-media to custom-media, to make it easier to grasp
---------
Co-authored-by: Ryan Johnson <escattone@gmail.com>
Prod:
Resolves#14553 Bump jinja2 from 3.1.3 to 3.1.4
Resolves#14532 Bump newrelic from 9.8.0 to 9.9.0
Resolves#14516 Bump boto3 from 1.34.79 to 1.34.101
Resolves#14526 Bump lxml from 5.2.0 to 5.2.1
Resolves#14528 Bump docutils from 0.20.1 to 0.21.2
Dev:
Resolves#14529 Bump pipdeptree from 2.17.0 to 2.19.1
Resolves#14524 Bump ruff from 0.3.5 to 0.4.2
Resolves#14514 Bump freezegun from 1.4.0 to 1.5.0
Docs:
Resolves#14520 Bump sphinx from 6.2.1 to 7.3.7
Resolves#14527 Bump sphinx-autobuild from 2021.3.14 to 2024.4.16
Resolves#14531 Bump myst-parser from 2.0.0 to 3.0.1
Subdeps, auto upgraded:
Resolves#14533 Bump referencing from 0.34.0 to 0.35.0
Resolves#14530 Bump botocore from 1.34.85 to 1.34.95
Resolves#14517 Bump jsonschema from 4.21.1 to 4.22.0
Closed - not doing these right now:
Resolves#14522 Bump django from 4.2.11 to 5.0.4
Resolves#14515 Bump tinycss2 from 1.2.1 to 1.3.0
Left open:
* #14521 Bump sentry-sdk from 1.44.0 to 2.0.1 - major release, see PR for link to steps
* #14519 Bump translate-toolkit from 3.12.2 to 3.13.0 - needs care/isolation/checking, due to dropped mozilla-related code
* #14525 Bump compare-locales from 9.0.2 to 9.0.3 - as above
* #14523 Bump pytest from 7.4.4 to 8.2.0 - needs upgrading in isolation with selenium in case requires rollback
* #14518 Bump selenium from 4.9.1 to 4.20.0 - see above
* Dependency bumps
**Prod deps**
Resolves#14420 Bump boto3 from 1.34.65 to 1.34.79
Resolves#14405 Bump pillow from 10.2.0 to 10.3.0
Resolves#14390 Bump lxml from 5.1.0 to 5.2.0
Resolves#14387 Bump sentry-sdk from 1.42.0 to 1.44.0
Resolves#14380 Bump pygithub from 2.2.0 to 2.3.0
Resolves#14379 Bump newrelic from 9.7.1 to 9.8.0
Resolves#14378 Bump jq from 1.6.0 to 1.7.0
**Dev deps**
Resolves#14422 Bump pipdeptree from 2.16.1 to 2.17.0
Resolves#14389 Bump pytest-mock from 3.12.0 to 3.14.0
Resolves#14385 Bump ruff from 0.3.3 to 0.3.5
Resolves#14381 Bump pytest-cov from 4.1.0 to 5.0.0
* Add management command to automatically refresh the root-cert policy doc
* First pass at updating the rootstore policy doc with the helper tool
* Remove Sentry-notification decorator, as it's irrelevant here
Rob Hudson
Alex Gibson
Steve Jalim
dependabot[bot]