In #14056 we standardized on requiring the "text" value to trigger link and cta click events
- Add "data-cta-text" values to tags with data-cta-type defined
- Fix#13749 where data attributes were not being output by Fluent templates
* Support sanitisation of Fluent strings before rendering
This changeset adds a layer of protection against unexpected markup being
introduced via Fluent strings as part of the localisation process.
We now use `bleach` to sanitise each Fluent string's content before declaring
it as safe for Django to render without escaping (via `Markup`).
The sanitisation only allows a specific, small, set of HTML tags and attributes
to be allowed through; everything else will be escaped.
We've drawn our initial set of allowed tags and attributes from the current
en-US fluent templates in the repo, following the logic that: a) we trust these
because we added them; and b) if someone tries to add in malevolent markup
via a Fluent string in a particular locale, we _should_ be sanitising that
because the translation should use the same tags and elements that the source
does.
There is a test added to show the sanitisation working, plus a test that
inspects the current state of the Fluent files in the l10n/ and l10n-pocket/
directories and ensures they do not contain any tags that are not currently
in the relevant allowlist. If we want to support another tag or attr in our
Fluent strings, that's fine - we just expand the allowlist, and a test will
fail to warn us that we haven't updated the allowlist.
* Add <i> to allowlist for fluent markup
I suspect this is transformed to <em> during some processing, but no harm including it as an allowed tag
* Switch test to use set for subset detection, to allow some allowed tags to be retired
* Update best locale match to return `None` instead of default.
* Connect list locale styles, add note to scss file
* Fixed tests
* Add no accept-language header tests
* Add contribute section at bottom of page
* Fix 500 error when `DEV` is set to `True`
* In careers tests set header on client
* Update copy to provide more context for locale-specific 404
* Remove bottom section, update copy
* Add fluent files, basic 404-locale debug view
* Update to load in ftl translations on 404-locale page
* Move /404-locale to only dev mode
* Fix functional tests by adding headers
* Make root path return 200 with alternate title
All three need upgrading at the same time because jinja 3.1+ removes deprecated
code which we were leaning on in Bedrock and django-jinja markdown - either
a convenience import of `Markup` or a change in decorators
* [Tiny] Improve error messaging if git repo interaction fails
* Override the Fluent repo-related config in Pocket mode
Note that for Pocket mode, we don't need an intermediary mozmeao/* repo to
hold the translations while we calculate activation metadata via a CI task
(which does happen for Mozorg). Why? All locales in Pocket should be 100%
ready to go, because they are translated by a vendor, whereas Mozorg has
community translation contributions as well, which aren't always exhaustive.
As a result, both FLUENT_REPO and FLUENT_L10N_TEAM_REPO point to the same repo
* Update Bedrock CI step to try to open a PR against the l10n repo
...if there are changes to the l10n files.
Note that we try to update both www-l10n (Mozorg) and pocket-www-l10n (Pocket)
each time it's run, with the plan that if it's run against an irrelevant repo
no changes will be detected, even though it's running twice
(This may need tweaking once it's running in CI properly, of course)
* Update settings.get_dev_languages to take params, to make re-use elsewhere easier
* Initial pass at setting up DEV and PROD languages for Pocket mode
* Ensure Pocket always uses all available production langs
All locales in Pocket should be 100% ready to go, because they are
translated by a vendor, whereas Mozorg has community translation
contributions as well, which aren't always exhaustive.
* Wind back "Update settings.get_dev_languages to take params, to make re-use elsewhere easier"
This reverts some of the work done in c653640ae56cf21e961c461a421d6dd5fa54efcd.
* Make DEV_LANGUAGES the same as PROD_LANGUAGES as there's no need not to
* Ensure Pocket-related l10n files are copied into Dockerimage during build
* Make the l10n_update command pull from both Mozorg and Pocket Fluent repos
* Split out pocket-l10n PR step in CI, so that Mozorg and Pocket steps run separately
* Update LANGUAGE_CODE for Pocket mode to be just 'en'
* Update output messaging to help keep track of FTL updates
* Nitfix: make env setting in Docker consistent
* Tidy up l10n-related settings following code review
* Nit: updating comments
* Fixup: Add quotes around bash variable substitution
* Tweak locale fallbacks
* Update locale setup for Spanish, but more work needed
Just noting some complexity to unpick here. getpocket.com/es-la/ exists, and
so does getpocket.com/es/. However it is sounding like /es/ needs to use
`es-ES` content from the vendor and /es-la/ needs to use `es` from the vendor.
Also note that Pocket currently uses `/es-la/` not `/es-LA/`
This all may require some extra wiring, as it feels like it's flipping over
the behaviour we have in bedrock.
* Move non-official es-la locale to FALLBACK_LOCALES
* Drop es-MX, es-CL, es-AR from FALLBACK_LOCALES - unnecessary
If the language_REGION locale string doesn't map to a specific locale, we try
to map the two-letter locale, so we don't need the variant locales for es.
* Drop unnecessary inclusion of en-* locale variants -- they'll fall back to /en/ fine without config
* Support running in Mozorg or Pocket modes in dev or prod via Docker, via Makefile commands
* Support Mozorg or Pocket mode via env var
Setting the SITE_MODE env var to Mozorg or Pocket will now switch the site
into serving only the URLs for that particular mode.
When in Pocket mode, its pages are served from the root path, not from
the 'externalpages' namespace/module name
The default behaviour, for now, remains as is: all URLs will be served, and
Pocket URLs are served as 'external/pocket/*`
* Flatten externalpages/pocket to just pocket
* Make Mozorg the default mode, with Pocket enabled via SITE_MODE env var
* Add in robots.txt for Pocket mode, based on current state of https://getpocket.com/robots.txt
* Revert "Add in robots.txt for Pocket mode, based on current state of https://getpocket.com/robots.txt"
This reverts commit 5934b4d613.
* Initial laying in of l10n-pocket directory and config files
* First pass at marking up a Pocket template for Fluent
There's more work to be done to wire it all up - the views aren't aware of the
Fluent side, yet
* Enable Fluent string replacement for Pocket About page
Note how we get the brands.ftl from main l10n to _always_ be used in Pocket
mode, regardless of locale.
* Switch to looking up Pocket paths via named URLs
* Fix slip in Fluent markup
* Update README for l10n-pocket with info about namespacing
* Markup spacing nitfix
* Improve markup of HTML: remove ids from strings; snake_case variables
* Switch BETA to title case, to avoid translation confusion with an acronym.
Has no semantic impact to change it.
* Fixup: two missing curly quotes
* Nitfix: snake_case for all FTL variables
* Fix up broken FTL markup: brands can't be used directly in a template; need to go via the template-specific FTL file
* Refactor attributes for hyperlinks to reduce risk of breakage and increaase scope to tune them without triggering new translations
* Collapse all press-link attrs into a single variable, to make less brittle for translation
* Add a Pocket-specific brands.ftl file and wire it in
The -brand-name-read-it-later-inc string is moved from l10n/en/brands.ftl
into the new Pocket-specific one, and can be used as proof that the new brands
file is being loaded, because the correct Read It Later, Inc string appears
at the bottom of the about page.
Also improve some of the brand-meets-product phrasing, to help with translation
- eg Pocket Premium, Pocket for Firefox
* Add 'es' locale, for mapping to non-Spain versions of Spanish
* Drop 'Pocket for Firefox' as a dedicated brand name -- unnecessary
* Ensure l10n-pocket/en/brands.ftl is disovered by default in Pocket mode
This means we can do without the override that forces the loading of brands
when in Pocket mode. We still need it for non-en locales, but at least now
it behaves the same as for Mozorg mode
* Update Pocket l10n README now the two modes do not share l10n files
* Typo fixup
* Explicitly redefine FLUENT_PATHS, to avoid pain later
This updates the `page` URL helper to wrap the view using the `require_safe`
decorator, and also limits the HTTP methods on the `L10nTemplateView` to `GET`
or `HEAD`. All other views explicitly get decorated with `@require_safe`.
* Remove all uses of gettext and `_()` and `_lazy()`.
* Remove all uses of `{% trans %}`
* Remove old l10n machinery
The uses of `{% trans %}` with variables, I've simply replaced many with `{% with %}` to minimize the changes.
* BUGFIX: always pop the preview_image key from the SEO object, so that a missing preview image doesn't break the sync
* 10867: Remove unique constraint on ContentfulEntry.contentful_id and make hybrid with locale field
This is required because translations of Entries in Contentful share the same ID but vary on locale, making the uniqueness check a combination of contentful_id and locale
* 10867: Move hard-coded IDs for Contentful Homepage lookup into settings, because we'll need them as part of some homepage-specific logic when syncing multuple locales very soon
* 10810: Log to Sentry if we end up using the static fallback homepages instead of data from Contentful
* 10867: Looking forward, support getting individual pages by ID and locale, not just ID.
This is backwards-compatible with the way that the EN and DE Homepages are pulled from the DB only via ID.
When we add in the multi-locale sync in an upcoming commit, we'll deliberately ONLY sync the homepages for en-US, because they don't really have a formal locale field set up, and we WON'T sync them for any other locale
* 10867: Add support for syncing multiple locales from Contentful
This changeset updates how we pull from Contentful to ensure we query for a
version of every page in every locale that's enabled in Contentful. It's not
the most efficient - there are quite a lost of wasted queries and we're not
getting a bunch of Entries at once, just iterating through N page * M locales,
but it's workable for now.
There are also a number of workarounds in the code:
1) When syncing the homepages (connectHomepage Entries) we know we can't
trust the locale field we're currently extracting for them, so we only
sync them for en-US. This will go away when we move the homepages to Compose.
2) When syncing a Compose `page` in any locale other than en-US, we get back
an Entry, but when we try to read its content, things blow up because there is
no child Entry linked to it. Because of this, we have to check for a specific
AttributeError message, and skip that page, but it does still cost us a network
call.
* 10867: Drop patching/truncation of es locales when going from Bedrock to Contentful because Contentful now has the variations of the locale
* 10867: Update tests for locale-supporting Contentful sync. More to come
* 10867: Reduce the noise during the sync
* 10867: Fix broken locale map from Bedrock to Contentful
🤦
* 10867: Constants cleanup while in the neighbourhood
* 10867: Update deletion logic so that removing an individual locale in Contentful works
The locale-specific version of the page is removed from the DB, but any others persist.
Before this change, the deleted locale persisted, which was not ideal even though the
use-case of removing just one locale from a published page is small
* Update comment to highlight risk of cacheing mutable data structures
* Rename test module for Contentful management commands
* 11022: Add helpers for deciding when we have enough Contentful content in a particular locale to 'activate' it.
Broadly similar to how we use Fluent files, we have a configurable threshold
percentage which we use to decide whether we have enough ContentfulEntry
records of the given classification and content_type in the target locale,
compared to the default locale, to say 'yes we have enough records to consider
this locale as active.
Use cases are building a list of active locales to use in the footer <select>
and also deciding if a listing page (eg the VRC page) is worth rendering in
a particular locale, or redirecting to the default locale]
* 11022: Refactor Contentful active_locales helper to hit the DB less
* Drop unnecessary cache-timeout config
* 10867: Fix typo in short locale code version of ja-JP
* 11022: [Almost] Guarantee that all locales will show content in DEV mode
This will overrire any threshold setting for locales for Contentful-sourced content
* 11022: Extend Contentful-specific active_locales helper with optional scoping for all content with a particular (shared) slug
* 11022: Update VRC listing and article views to support multiple locales
* If requested locale isn't available for a specific article, try defaulting to en-US for the same slug, else 404
* Show the locale selector on listing and article pages, showing only those locales that are 'active', as defined by the helper util from an earlier commit
* Backfill tests for ContentfulEntry model - now at 100%
* Test coverage papercuts for Contentful management command
* Add pytest-mock to dev deps
* Contentful sync: split up main method to be a bit smaller + add another coverage increment
* Test regression fixups after rebasing
* Update comment on Bedrock<->Contentful locale map to help avoid misunderstanding
* VRC listing view: use l10n_utils.render to automatically select an appropriate locale if selected one is inviable
Returning early with the call to render() is better than a forced redirect to a DEFAULT_LOCALE, because it will
try to get the most appropriate default locale it can, rather than assuming en-US.
* Rename helper function used to transform Contentful locale names to Bedrock locale names
* VRC article view: use l10n_utils.render to automatically select an appropriate locale
...if the selected one is inviable
* Trim comments
* Optimise get_active_locales: return early when we've got a slug in the kwargs
When finding locales by slug + content_type + classification, every result returned
is active 'enough' because we only need one match for that triplet of params for it
to count as 'enough'. As such, we can return early and avoid crunching data unnecessarily
* Expand tests to cover both unavailable and invalid locales
* Refactor the redirection logic from l10n_utils.render() into a reusable function
Tested manually with DEV=False to be able to replicate locale redirection for /ff/ as a locale
* Refactor VRC to use redirect_to_best_locale helper - much cleaner now
* Tidy up the dev-level override for the Contentful locale activation threshold
* 10614: Add pre-commit hook for including the MPLv2
Adds to Python, JS, SCSS, Jinja HTML, Fluent templates and shell scripts
Note that the order of application of the hooks is important - we want to add a missing license before we check the formatting of files
* 10614: Update MPLv2 comments on all templates to match standard format produced by pre-commit hook
* Updates existing MPLv2 text to use a https URL
* Amend a handful of Fluent templates that used a token instead of the string "Mozilla" - this standardised things; translation was not used or needed
* Add missing MPLv2 where needed
* Update three tests that regressed with these changes, above
* 10614: Update pre-commit config to not add MPL to JS libraries; Remove MPL from the four files which should not have had it
* Update requirements to avoid requests version clash
* 10570: Add isort to project
* 10570: Add isort config to project
* 10570: Add isort as a test run/CI step
* 10570: Add isort to pre-commit config
* 10570: Update isort config to allow tests/pages.py to be first-party, which is more appropriate
* 10570: Apply isort to entire codebase, using project config
* 10570: Explicitly include lib/ as a first-party import
This behaviour is already implicit in the current config, but making it explicit, partly to protect against a future change. Also makes it easier to grok what will happen to various bedrock-codebase imports
* 10570: Update isort config to stack Django imports above third-party ones
* 10570: [nit-fix] Update run-tests.sh to call isort with the same syntax pattern as we call black
* Convert English and German home pages to use Contentful (behind a
switch)
* Add docs for Contentful integration
Fix#9990#9991#9480#10268
Co-authored-by: Paul McLanahan <pmac@mozilla.com>
Automation currently always opens a new PR for FTL file updates. This
checks for an open PR and updates the branch instead of creating a new
PR everytime, since the new PR will contain changes from the other open
one.
It was caching the `FluentLocalization` object, but that caused issues
in multithreaded environments (like the development server) due to
simultaneous attempts at iteration. This change caches loading the
Fluent resources which is the expensive work of parsing the files
and bundling the results into lists of resources.
Fix#8705
Stephanie Hobson
Rob Hudson
Nathan Barrett
Reem H
Steve Jalim
Paul McLanahan
Paul McLanahan