* Allow disabling Markus when DEBUG=True (default: keep it enabled)
* Add Wagtail 5.2 LTS as a dependency
* Hook in Wagtail Admin, while also keeping jinja-template rendering happy
* Support user-uploaded images into the CMS, both with local and cloud storage options
Of course, the cloud storage option is the only one we'll use when deployed.
Note that this also disables CSP for any routes inside the Wagtail admin
(but not pages rendered by Wagtail)
* Drop in wagtail-localize, ready for i18n/l10n
Also fix up some test regressions as a result of this change, including
changing how Careers renders a custom 404-page message
* Lay in intial 'cms' app as home for our Wagtail-related modelling.
* Support detection of CMS-based pages so that we don't falsely redirect away from pages managed by wagtail-localize
* Wrap simple CMS test page in a Protocol-based base template to confirm we load the right strings and get the lang picker
* Ensure that pages fall back (in their enriety) if we don't have a page with the matching lang code in the CMS
* Exclude the cms-admin path from the sitemap - doesn't need to be discoverable
* Remove now-redundant print statement from a data migration
* Add tests to show locales from page models are used as translations for pages
* Add tests to show that pages with restrictions (e.g. passwords) should not be cached downstream
* Add tests for StructuralPage, confirming it redirects to its parent if accessed directly
* Expand test coverage to include serve_preview for our base page model
* Fixup accidentally-commented-out app in INSTALLED_APPS
Co-authored-by: Ryan Johnson <escattone@gmail.com>
* Improve comment related to local GCS credentials, clarifying who will/won't need them and why
* Disable use of Gravatar URLs for Wagtail users
Low benefit to us, because they need CSP tweaks and also would need scrubbing before the DB export.
* Fix labelling of Wagtail's homepage
* We don't want wagtailforms enabled in our setup: public bedrock is read only
* Back out hack to 404.html for careers-related 404 -- unnecessary following i18n/routing refactor
* Back out unnecessary follow=True on the 404 test
* Ensure that demos run migrations on their local DB, so that any new Wagtail tables get made ASAP, preventing 500s
* Add a fake spec for WAGTAIL_CONTENT_LANGUAGES in Pocket mode, so that it'll boot.
We won't be using wagtail for Pocket mode
* Rename MEDIA_URL from user-media to custom-media, to make it easier to grasp
---------
Co-authored-by: Ryan Johnson <escattone@gmail.com>
* [Breaking] Drop our custom locale middleware and switch to Django's
* [Breaking] Allow mixed-case lang codes as settings.LANGUAGES and related lang-code lookups
* [Breaking] Use i18n_patterns to lang-code prefix relevant paths in mozorg.urls
This includes splitting mozorg.urls into two groups so we only use
i18n_patterns with paths that need it. We've kept the main, localed, group's
name to be urls.py to reduce the chance of merge conflicts, as this piece
of work is likely to need rebasing against main quite a lot until it's ready.
The decision about which get prefixed and which do not is based on the
settings SUPPORTED_NONLOCALES and SUPPORTED_LOCALE_IGNORE that were used
in our former i18n machinery. At the time of committing, they still exist,
but both of should be able to be removed once this work is done (or at least
moved to power some tests)
* [Breaking] We now have /en-US/ resolving/rendering for the homepage, but not yet /fr/
We use a prefixer for our i18n_urlpatterns so that when we reverse() and
resolve() we allow for a custom prefix that is normalized to our mixed-case
lang codes.
Tests are needed.
* Add middleware to fix up the lang code in the path
This essentially replaces urlresolves.Prefixer.fix()
Tests are needed.
* Lay in custom middleware that lightly wraps LocaleMiddleware to normalize lang codes
* Switch to using django.utils.translation to track the active locale
* Ensure the request is annotated with the relevant lang_code, so that it's
available in templates
* Refactor base test class with simpler locale-activation logic using django core i18n
* Minor fix to modern way to call pytest
* Fix serving of sitemap_none.xml to be without locale
* Small-as-possible changes to get test suite running/not erroring
* Fixup VPN Resource Centre tests - enabling a non-real locale was trickiest part
* Update tests for accepted locales
Note that we no longer have a lowercase-to-mixed-case lang lookup, so this
test changes but might become redundant later in the refactor
* Improve naming in helper
* [Breaking] Add placeholder failing tests
* Extend Django's support for get_language_info to include our three-character lang names, which it doesn't have by default
This is a part of ensuring three-letter country codes still work, but it might
be redundant - TBC
* Use Django's bidi-lang-checker, not ours
* Drop unnecessary work in custom locale middleware
This was useful before we started using i18n_patterns, but now not all of it
is needed. Some still is, though
* Fixup redirects behaviour (using /webvision/ as an example case)
* Redirects don't need url prefixing via i18n_patterns, so we're moving them to
nonlocale_urls instead.
* When detecting a language, en-CA was being picked over en-US, which is
avoided by ensuring en-US is at the start of the languages list
* Note that for the webvision redirect we needed to add:
* locale_prefix=True
* prepend_locale=False
* Amend VPNRC tests - tweaking where appropriate; also dropped a fiddly and unrealistic test
* Expand test coverage of redirects in bedrock.mozorg
* Fix need for fallback to settings.LANGUAGE_CODE if no language can be detected/retrieved
* Reinstate logic to handle fallback to appropriate locale based on Accept-Language header
* Move mozorg redirect() calls to redirects.py to avoid the routes having lang codes prefixed to them
* Fixup l10n_utils.render behaviour via minimal tweaks to tests
* Add regression-check tests for utility views
* Improve normalize_locale via tests, including tests for Pocket Mode
* Add tests for i18n helpers
* Improve language normalization + tests
* Move careers redirects to own file, so that they remain lang-code agnostic
* Move foundation redirect out of urls.py so that it remains language agnostic
* Move redirects out of urls.py so that they remain language agnostic - tweak _one_ test related to this
* Avoid unnecessarily adding lang code to a reversed() url in our redirector - we don't need it unless we explicitly ask for it
* Fix lang codes for CANONICAL_LANGS for mozorg mode
* Tidy up tests
* Fixing tests with settings=DEV
* Contentful-related tests needed the dev-urls config wrapping in URLpatterns
to make it work as per the real site; if we just specced dev_urls they did not
work because they were not i18n-patterned
* Default a locale to '' not None - the latter causes Locale page to blow up and the former matches original behaviour better
* Ensure no-JS language changing works
Fixes#5931
See https://github.com/mozilla/bedrock/issues/5931 for context
* Formatting fix after rebasing on main
* Rename our get_language helper to make it clearer what it does
* Monkeypatch Django's check_for_language() because it looks in the wrong place for us
By default check_for_language() seeks gettext-compatible files in locale/ directories
in the project, but we don't have those because our l10n is handled using Fluent
strings in the www-l10n-team directory, cloned from their git repo.
So, instead, we just perform a light check to see whether a given lang is among the
langs that shold be available. The actual decision about whether we have have (enough)
Fluent strings available for that locale happens in lib.l10n_utils.render()
* Expand test coverage of our custom prefixer
* Bring bedrock.base.i18n up to 100% coverage
* Add tests for language-code fixup + slight refactor to ensure it doesn't to _too_ much
* Fix failing startup check for Pocket mode that blocked everything running
* Drop unnecessary-after-all patching of LANG_INFO - we found a different route to making check_for_language work; this was the wrong direction
* Add a little extra logging to the language-fixup redirects we're doing
* Minor test polishing
* Support special-case showing of a locale list at / if no accept-lang header is present
This changeset ensures we retain the existing behaviour where a robot/spider
going to / with no accept-language header gets a 200 OK response but with a
locale-picker page, NOT the homepage.
The fix here was to avoid running Django's LocaleMiddleware when (and only
when) we detect the special case of a request for the root path and no
Accept-Language header - which is basically a call for the root page with
no way to determine the client's locale. (We don't use sessions in Bedrock
so Django will never find a language code in a session anyway)
* Improve naming to avoid confusion between polish (verb) and Polish in split_path_and_polish_lang
* Improve in-code docs for BedrockLangCodeFixupMiddleware
* Test tweak after rebasing on main (which now has Django 4.2)
In #14056 we standardized on requiring the "text" value to trigger link and cta click events
- Add "data-cta-text" values to tags with data-cta-type defined
- Fix#13749 where data attributes were not being output by Fluent templates
* Support sanitisation of Fluent strings before rendering
This changeset adds a layer of protection against unexpected markup being
introduced via Fluent strings as part of the localisation process.
We now use `bleach` to sanitise each Fluent string's content before declaring
it as safe for Django to render without escaping (via `Markup`).
The sanitisation only allows a specific, small, set of HTML tags and attributes
to be allowed through; everything else will be escaped.
We've drawn our initial set of allowed tags and attributes from the current
en-US fluent templates in the repo, following the logic that: a) we trust these
because we added them; and b) if someone tries to add in malevolent markup
via a Fluent string in a particular locale, we _should_ be sanitising that
because the translation should use the same tags and elements that the source
does.
There is a test added to show the sanitisation working, plus a test that
inspects the current state of the Fluent files in the l10n/ and l10n-pocket/
directories and ensures they do not contain any tags that are not currently
in the relevant allowlist. If we want to support another tag or attr in our
Fluent strings, that's fine - we just expand the allowlist, and a test will
fail to warn us that we haven't updated the allowlist.
* Add <i> to allowlist for fluent markup
I suspect this is transformed to <em> during some processing, but no harm including it as an allowed tag
* Switch test to use set for subset detection, to allow some allowed tags to be retired
* Update best locale match to return `None` instead of default.
* Connect list locale styles, add note to scss file
* Fixed tests
* Add no accept-language header tests
* Add contribute section at bottom of page
* Fix 500 error when `DEV` is set to `True`
* In careers tests set header on client
* Update copy to provide more context for locale-specific 404
* Remove bottom section, update copy
* Add fluent files, basic 404-locale debug view
* Update to load in ftl translations on 404-locale page
* Move /404-locale to only dev mode
* Fix functional tests by adding headers
* Make root path return 200 with alternate title
All three need upgrading at the same time because jinja 3.1+ removes deprecated
code which we were leaning on in Bedrock and django-jinja markdown - either
a convenience import of `Markup` or a change in decorators
* [Tiny] Improve error messaging if git repo interaction fails
* Override the Fluent repo-related config in Pocket mode
Note that for Pocket mode, we don't need an intermediary mozmeao/* repo to
hold the translations while we calculate activation metadata via a CI task
(which does happen for Mozorg). Why? All locales in Pocket should be 100%
ready to go, because they are translated by a vendor, whereas Mozorg has
community translation contributions as well, which aren't always exhaustive.
As a result, both FLUENT_REPO and FLUENT_L10N_TEAM_REPO point to the same repo
* Update Bedrock CI step to try to open a PR against the l10n repo
...if there are changes to the l10n files.
Note that we try to update both www-l10n (Mozorg) and pocket-www-l10n (Pocket)
each time it's run, with the plan that if it's run against an irrelevant repo
no changes will be detected, even though it's running twice
(This may need tweaking once it's running in CI properly, of course)
* Update settings.get_dev_languages to take params, to make re-use elsewhere easier
* Initial pass at setting up DEV and PROD languages for Pocket mode
* Ensure Pocket always uses all available production langs
All locales in Pocket should be 100% ready to go, because they are
translated by a vendor, whereas Mozorg has community translation
contributions as well, which aren't always exhaustive.
* Wind back "Update settings.get_dev_languages to take params, to make re-use elsewhere easier"
This reverts some of the work done in c653640ae56cf21e961c461a421d6dd5fa54efcd.
* Make DEV_LANGUAGES the same as PROD_LANGUAGES as there's no need not to
* Ensure Pocket-related l10n files are copied into Dockerimage during build
* Make the l10n_update command pull from both Mozorg and Pocket Fluent repos
* Split out pocket-l10n PR step in CI, so that Mozorg and Pocket steps run separately
* Update LANGUAGE_CODE for Pocket mode to be just 'en'
* Update output messaging to help keep track of FTL updates
* Nitfix: make env setting in Docker consistent
* Tidy up l10n-related settings following code review
* Nit: updating comments
* Fixup: Add quotes around bash variable substitution
* Tweak locale fallbacks
* Update locale setup for Spanish, but more work needed
Just noting some complexity to unpick here. getpocket.com/es-la/ exists, and
so does getpocket.com/es/. However it is sounding like /es/ needs to use
`es-ES` content from the vendor and /es-la/ needs to use `es` from the vendor.
Also note that Pocket currently uses `/es-la/` not `/es-LA/`
This all may require some extra wiring, as it feels like it's flipping over
the behaviour we have in bedrock.
* Move non-official es-la locale to FALLBACK_LOCALES
* Drop es-MX, es-CL, es-AR from FALLBACK_LOCALES - unnecessary
If the language_REGION locale string doesn't map to a specific locale, we try
to map the two-letter locale, so we don't need the variant locales for es.
* Drop unnecessary inclusion of en-* locale variants -- they'll fall back to /en/ fine without config
* Support running in Mozorg or Pocket modes in dev or prod via Docker, via Makefile commands
* Support Mozorg or Pocket mode via env var
Setting the SITE_MODE env var to Mozorg or Pocket will now switch the site
into serving only the URLs for that particular mode.
When in Pocket mode, its pages are served from the root path, not from
the 'externalpages' namespace/module name
The default behaviour, for now, remains as is: all URLs will be served, and
Pocket URLs are served as 'external/pocket/*`
* Flatten externalpages/pocket to just pocket
* Make Mozorg the default mode, with Pocket enabled via SITE_MODE env var
* Add in robots.txt for Pocket mode, based on current state of https://getpocket.com/robots.txt
* Revert "Add in robots.txt for Pocket mode, based on current state of https://getpocket.com/robots.txt"
This reverts commit 5934b4d613.
* Initial laying in of l10n-pocket directory and config files
* First pass at marking up a Pocket template for Fluent
There's more work to be done to wire it all up - the views aren't aware of the
Fluent side, yet
* Enable Fluent string replacement for Pocket About page
Note how we get the brands.ftl from main l10n to _always_ be used in Pocket
mode, regardless of locale.
* Switch to looking up Pocket paths via named URLs
* Fix slip in Fluent markup
* Update README for l10n-pocket with info about namespacing
* Markup spacing nitfix
* Improve markup of HTML: remove ids from strings; snake_case variables
* Switch BETA to title case, to avoid translation confusion with an acronym.
Has no semantic impact to change it.
* Fixup: two missing curly quotes
* Nitfix: snake_case for all FTL variables
* Fix up broken FTL markup: brands can't be used directly in a template; need to go via the template-specific FTL file
* Refactor attributes for hyperlinks to reduce risk of breakage and increaase scope to tune them without triggering new translations
* Collapse all press-link attrs into a single variable, to make less brittle for translation
* Add a Pocket-specific brands.ftl file and wire it in
The -brand-name-read-it-later-inc string is moved from l10n/en/brands.ftl
into the new Pocket-specific one, and can be used as proof that the new brands
file is being loaded, because the correct Read It Later, Inc string appears
at the bottom of the about page.
Also improve some of the brand-meets-product phrasing, to help with translation
- eg Pocket Premium, Pocket for Firefox
* Add 'es' locale, for mapping to non-Spain versions of Spanish
* Drop 'Pocket for Firefox' as a dedicated brand name -- unnecessary
* Ensure l10n-pocket/en/brands.ftl is disovered by default in Pocket mode
This means we can do without the override that forces the loading of brands
when in Pocket mode. We still need it for non-en locales, but at least now
it behaves the same as for Mozorg mode
* Update Pocket l10n README now the two modes do not share l10n files
* Typo fixup
* Explicitly redefine FLUENT_PATHS, to avoid pain later
This updates the `page` URL helper to wrap the view using the `require_safe`
decorator, and also limits the HTTP methods on the `L10nTemplateView` to `GET`
or `HEAD`. All other views explicitly get decorated with `@require_safe`.
* Remove all uses of gettext and `_()` and `_lazy()`.
* Remove all uses of `{% trans %}`
* Remove old l10n machinery
The uses of `{% trans %}` with variables, I've simply replaced many with `{% with %}` to minimize the changes.
* BUGFIX: always pop the preview_image key from the SEO object, so that a missing preview image doesn't break the sync
* 10867: Remove unique constraint on ContentfulEntry.contentful_id and make hybrid with locale field
This is required because translations of Entries in Contentful share the same ID but vary on locale, making the uniqueness check a combination of contentful_id and locale
* 10867: Move hard-coded IDs for Contentful Homepage lookup into settings, because we'll need them as part of some homepage-specific logic when syncing multuple locales very soon
* 10810: Log to Sentry if we end up using the static fallback homepages instead of data from Contentful
* 10867: Looking forward, support getting individual pages by ID and locale, not just ID.
This is backwards-compatible with the way that the EN and DE Homepages are pulled from the DB only via ID.
When we add in the multi-locale sync in an upcoming commit, we'll deliberately ONLY sync the homepages for en-US, because they don't really have a formal locale field set up, and we WON'T sync them for any other locale
* 10867: Add support for syncing multiple locales from Contentful
This changeset updates how we pull from Contentful to ensure we query for a
version of every page in every locale that's enabled in Contentful. It's not
the most efficient - there are quite a lost of wasted queries and we're not
getting a bunch of Entries at once, just iterating through N page * M locales,
but it's workable for now.
There are also a number of workarounds in the code:
1) When syncing the homepages (connectHomepage Entries) we know we can't
trust the locale field we're currently extracting for them, so we only
sync them for en-US. This will go away when we move the homepages to Compose.
2) When syncing a Compose `page` in any locale other than en-US, we get back
an Entry, but when we try to read its content, things blow up because there is
no child Entry linked to it. Because of this, we have to check for a specific
AttributeError message, and skip that page, but it does still cost us a network
call.
* 10867: Drop patching/truncation of es locales when going from Bedrock to Contentful because Contentful now has the variations of the locale
* 10867: Update tests for locale-supporting Contentful sync. More to come
* 10867: Reduce the noise during the sync
* 10867: Fix broken locale map from Bedrock to Contentful
🤦
* 10867: Constants cleanup while in the neighbourhood
* 10867: Update deletion logic so that removing an individual locale in Contentful works
The locale-specific version of the page is removed from the DB, but any others persist.
Before this change, the deleted locale persisted, which was not ideal even though the
use-case of removing just one locale from a published page is small
* Update comment to highlight risk of cacheing mutable data structures
* Rename test module for Contentful management commands
* 11022: Add helpers for deciding when we have enough Contentful content in a particular locale to 'activate' it.
Broadly similar to how we use Fluent files, we have a configurable threshold
percentage which we use to decide whether we have enough ContentfulEntry
records of the given classification and content_type in the target locale,
compared to the default locale, to say 'yes we have enough records to consider
this locale as active.
Use cases are building a list of active locales to use in the footer <select>
and also deciding if a listing page (eg the VRC page) is worth rendering in
a particular locale, or redirecting to the default locale]
* 11022: Refactor Contentful active_locales helper to hit the DB less
* Drop unnecessary cache-timeout config
* 10867: Fix typo in short locale code version of ja-JP
* 11022: [Almost] Guarantee that all locales will show content in DEV mode
This will overrire any threshold setting for locales for Contentful-sourced content
* 11022: Extend Contentful-specific active_locales helper with optional scoping for all content with a particular (shared) slug
* 11022: Update VRC listing and article views to support multiple locales
* If requested locale isn't available for a specific article, try defaulting to en-US for the same slug, else 404
* Show the locale selector on listing and article pages, showing only those locales that are 'active', as defined by the helper util from an earlier commit
* Backfill tests for ContentfulEntry model - now at 100%
* Test coverage papercuts for Contentful management command
* Add pytest-mock to dev deps
* Contentful sync: split up main method to be a bit smaller + add another coverage increment
* Test regression fixups after rebasing
* Update comment on Bedrock<->Contentful locale map to help avoid misunderstanding
* VRC listing view: use l10n_utils.render to automatically select an appropriate locale if selected one is inviable
Returning early with the call to render() is better than a forced redirect to a DEFAULT_LOCALE, because it will
try to get the most appropriate default locale it can, rather than assuming en-US.
* Rename helper function used to transform Contentful locale names to Bedrock locale names
* VRC article view: use l10n_utils.render to automatically select an appropriate locale
...if the selected one is inviable
* Trim comments
* Optimise get_active_locales: return early when we've got a slug in the kwargs
When finding locales by slug + content_type + classification, every result returned
is active 'enough' because we only need one match for that triplet of params for it
to count as 'enough'. As such, we can return early and avoid crunching data unnecessarily
* Expand tests to cover both unavailable and invalid locales
* Refactor the redirection logic from l10n_utils.render() into a reusable function
Tested manually with DEV=False to be able to replicate locale redirection for /ff/ as a locale
* Refactor VRC to use redirect_to_best_locale helper - much cleaner now
* Tidy up the dev-level override for the Contentful locale activation threshold
* 10614: Add pre-commit hook for including the MPLv2
Adds to Python, JS, SCSS, Jinja HTML, Fluent templates and shell scripts
Note that the order of application of the hooks is important - we want to add a missing license before we check the formatting of files
* 10614: Update MPLv2 comments on all templates to match standard format produced by pre-commit hook
* Updates existing MPLv2 text to use a https URL
* Amend a handful of Fluent templates that used a token instead of the string "Mozilla" - this standardised things; translation was not used or needed
* Add missing MPLv2 where needed
* Update three tests that regressed with these changes, above
* 10614: Update pre-commit config to not add MPL to JS libraries; Remove MPL from the four files which should not have had it
* Update requirements to avoid requests version clash
* 10570: Add isort to project
* 10570: Add isort config to project
* 10570: Add isort as a test run/CI step
* 10570: Add isort to pre-commit config
* 10570: Update isort config to allow tests/pages.py to be first-party, which is more appropriate
* 10570: Apply isort to entire codebase, using project config
* 10570: Explicitly include lib/ as a first-party import
This behaviour is already implicit in the current config, but making it explicit, partly to protect against a future change. Also makes it easier to grok what will happen to various bedrock-codebase imports
* 10570: Update isort config to stack Django imports above third-party ones
* 10570: [nit-fix] Update run-tests.sh to call isort with the same syntax pattern as we call black
* Convert English and German home pages to use Contentful (behind a
switch)
* Add docs for Contentful integration
Fix#9990#9991#9480#10268
Co-authored-by: Paul McLanahan <pmac@mozilla.com>
Rob Hudson
Jan Brasna
Steve Jalim
Stephanie Hobson
Nathan Barrett
Reem H
Paul McLanahan
Paul McLanahan