mozilla
/
bedrock
зеркало из https://github.com/mozilla/bedrock.git
1
0
Форкнуть 0
Код Задачи Пакеты Проекты Релизы Вики Активность

Contentful: Migrate away from legacy Compose (#12133) 

* Add set of Contentful-related migrations that migrate us away from the
"Legacy Compose" pattern. This is based on
https://www.contentful.com/developers/docs/compose/upgrade-to-customizable-compose-content-model/
plus investigations by @maureenlholland.

The migrations are kept separate so they can be applied individually and
progress monitored. Also, it makes sense to 'commit' each step separately,
rather than risk locking/race issues with the Contentful API's datastore.
(This is an assumption on my part - maybe we _could_ do all this in one go.)

First one: Adds new fields to any page type that previously used Legacy Compose
(In our case that's just the pagePageResourceCenter content type.)

Second one: Populates data, drawing from the Compose:Page entry and setting it
on the pagePageResourceCenter entries.

Third one: cleanup, unpublishing and deleting the Compose:Page entries and then
removing it as a content type entirely

* Update backend Python code to match updated page Schema now we have dropped legacy Compose

Note that the Connect:Homepage approach still leaves us some niggles that
we should ideally get past soon, too

* Improve ordering of fields for pagePageResourceCenter via Contentful migration

Example of it running:

Update Content Type pagePageResourceCenter

  Move field title after field name

  Move field slug after field title

  Move field seo after field slug

Publish Content Type pagePageResourceCenter
Update Content Type pagePageResourceCenter
Migration successful

* Add ADR for how we're handling Contentful migrations

* Update README for Contentful migrations approach

* Update README and ADR with typo fixes

* Typo fixups, following code review

* Add Contentful migration to make Title and Slug on Resource Center pages required

* Add Contentful migration to force-publish all Resource Center pages
This commit is contained in:
Steve Jalim 2022-11-02 15:26:12 +00:00 коммит произвёл GitHub
Родитель 6becad8e00
Коммит 131c6b2a3b
Не найден ключ, соответствующий данной подписи
Идентификатор ключа GPG: 4AEE18F83AFDEB23
22 изменённых файлов: 4484 добавлений и 125 удалений
Показать статистику Скачать Patch файл Скачать Diff файл Показать все файлы Свернуть все файлы

2
.editorconfig
Просмотреть файл

@ -10,7 +10,7 @@ insert_final_newline = true
trim_trailing_whitespace = true
indent_style = space
[*.{py,js,css,scss,sh,groovy}]
[*.{py,js,cjs,css,scss,sh,groovy}]
indent_size = 4
[*.{json,html,svg,yml}]

3
.eslintrc.js
Просмотреть файл

@ -3,13 +3,14 @@
* License, v. 2.0. If a copy of the MPL was not distributed with this
* file, You can obtain one at https://mozilla.org/MPL/2.0/.
*/
/* eslint-env es6 */
module.exports = {
env: {
browser: true,
commonjs: true
},
extends: ['eslint:recommended', 'plugin:json/recommended', 'prettier'],
ignorePatterns: ['contentful_migrations/migrations/*.cjs'],
rules: {
// Require strict mode directive in top level functions
// https://eslint.org/docs/rules/strict

30
bedrock/contentful/api.py
Просмотреть файл

@ -17,7 +17,6 @@ from rich_text_renderer.block_renderers import BaseBlockRenderer
from rich_text_renderer.text_renderers import BaseInlineRenderer
from bedrock.contentful.constants import (
COMPOSE_MAIN_PAGE_TYPE,
CONTENT_TYPE_PAGE_GENERAL,
CONTENT_TYPE_PAGE_RESOURCE_CENTER,
)
@ -479,17 +478,8 @@ class ContentfulPage:
return f"https:{preview_image_url}"
def _get_info_data__slug_title_blurb(self, entry_fields, seo_fields):
if self.page.content_type.id == COMPOSE_MAIN_PAGE_TYPE:
# This means we're dealing with a Compose-structured setup,
# and the slug lives not on the Entry, nor the SEO object
# but just on the top-level Compose `page`
slug = self.page.fields().get("slug")
else:
# Non-Compose pages
slug = entry_fields.get("slug", "home") # TODO: check if we can use a better fallback
title = getattr(self.page, "title", "")
slug = entry_fields.get("slug", "home") # TODO: check if we can use a better fallback
title = entry_fields.get("title", "")
title = entry_fields.get("preview_title", title)
blurb = entry_fields.get("preview_blurb", "")
@ -542,7 +532,6 @@ class ContentfulPage:
return {"locale": locale}
def get_info_data(self, entry_obj, seo_obj=None):
# TODO, need to enable connectors
entry_fields = entry_obj.fields()
if seo_obj:
seo_fields = seo_obj.fields()
@ -591,20 +580,15 @@ class ContentfulPage:
return data
def get_content(self):
# Check if it is a page or a connector, or a Compose page type
# Check if it is a page or a connector
entry_type = self.page.content_type.id
seo_obj = None
if entry_type == COMPOSE_MAIN_PAGE_TYPE:
# Contentful Compose page, linking to content and SEO models
entry_obj = self.page.content # The page with the actual content
seo_obj = self.page.seo # The SEO model
# Note that the slug lives on self.page, not the seo_obj.
elif entry_type.startswith("page"):
entry_obj = self.page
elif entry_type == "connectHomepage":
if entry_type == "connectHomepage":
# Legacy - TODO: remove me once we're no longer using Connect: Homepage
entry_obj = self.page.fields()["entry"]
elif entry_type.startswith("page"): # WARNING: this requires a consistent naming of page types in Contentful, too
entry_obj = self.page
seo_obj = self.page.seo
else:
raise ValueError(f"{entry_type} is not a recognized page type")

13
bedrock/contentful/constants.py
Просмотреть файл

@ -2,11 +2,6 @@
# License, v. 2.0. If a copy of the MPL was not distributed with this
# file, You can obtain one at https://mozilla.org/MPL/2.0/.
# Which models do we want to sync?
COMPOSE_MAIN_PAGE_TYPE = "page"
MAX_MESSAGES_PER_QUEUE_POLL = 10
# Specific content types we need to target in DB lookups
@ -15,15 +10,13 @@ CONTENT_TYPE_PAGE_RESOURCE_CENTER = "pagePageResourceCenter"
CONTENT_TYPE_PAGE_GENERAL = "pageGeneral"
DEFAULT_CONTENT_TYPES = ",".join(
# Soon, we'll only need to sync the Compose-driven `page` type, but
# until then we also still have homepages set with the Connect pattern
[
CONTENT_TYPE_CONNECT_HOMEPAGE, # The Connect-based approach, used for the homepage
COMPOSE_MAIN_PAGE_TYPE, # General Compose Page type - the related `content` type's name is what we store in the DB
CONTENT_TYPE_CONNECT_HOMEPAGE, # The Connect-based approach, currently used for the homepage
CONTENT_TYPE_PAGE_RESOURCE_CENTER, # New-era Compose page with a dedicated type
]
)
CONTENT_CLASSIFICATION_VPN = "VPN" # Matches string in Contenful for VPN as `product`
CONTENT_CLASSIFICATION_VPN = "VPN" # Matches string in Contentful for VPN as `product`
ARTICLE_CATEGORY_LABEL = "category" # for URL-param filtering

8
bedrock/contentful/management/commands/update_contentful.py
Просмотреть файл

@ -25,7 +25,6 @@ from bedrock.contentful.constants import (
ACTION_SAVE,
ACTION_UNARCHIVE,
ACTION_UNPUBLISH,
COMPOSE_MAIN_PAGE_TYPE,
CONTENT_TYPE_CONNECT_HOMEPAGE,
MAX_MESSAGES_PER_QUEUE_POLL,
)
@ -362,13 +361,6 @@ class Command(BaseCommand):
error_count += 1
continue
# Compose-authored pages have a page_type of `page`
# but really we want the entity the Compose page references
if ctype == COMPOSE_MAIN_PAGE_TYPE:
# TODO: make this standard when we _only_ have Compose pages,
# because they all have a parent type of COMPOSE_MAIN_PAGE_TYPE
ctype = page_data["page_type"]
hash = data_hash(page_data)
_info = page_data["info"]

2
bedrock/contentful/models.py
Просмотреть файл

@ -60,7 +60,7 @@ class ContentfulEntryManager(models.Manager):
order_by (str, optional): Sorting key for the queryset. Defaults to "last_modified".
Returns:
QuerySet[ContentfulEntry]: the main ContenfulEntry models, not just their JSON data
QuerySet[ContentfulEntry]: the main ContentfulEntry models, not just their JSON data
"""
kwargs = dict(

76
bedrock/contentful/tests/test_contentful_api.py
Просмотреть файл

@ -43,7 +43,6 @@ from bedrock.contentful.api import (
get_client,
)
from bedrock.contentful.constants import (
COMPOSE_MAIN_PAGE_TYPE,
CONTENT_TYPE_CONNECT_HOMEPAGE,
CONTENT_TYPE_PAGE_RESOURCE_CENTER,
)
@ -681,7 +680,7 @@ def test__render_list():
@pytest.fixture
def basic_contentful_page(rf):
"""Naive reusable fixutre for setting up a ContentfulPage
"""Naive reusable fixture for setting up a ContentfulPage
Note that it does NOTHING with set_current_request / thread-locals
"""
with patch("bedrock.contentful.api.set_current_request"):
@ -868,84 +867,90 @@ def test_ContentfulPage__get_info_data__locale(
@pytest.mark.parametrize(
"page_title, page_type, page_fields, entry_fields, seo_fields, expected",
"page_type, legacy_connect_page_fields, entry_fields, seo_fields, expected",
(
(
"test page one",
COMPOSE_MAIN_PAGE_TYPE,
{"slug": "compose-main-page-slug"},
{},
{},
CONTENT_TYPE_PAGE_RESOURCE_CENTER,
{}, # Connect-type fields
{
"slug": "compose-main-page-slug",
"slug": "vrc-main-page-slug",
"title": "test page one",
}, # fields from the page itself
{}, # SEO object's fields
{
"slug": "vrc-main-page-slug",
"title": "test page one",
"blurb": "",
},
),
(
"",
COMPOSE_MAIN_PAGE_TYPE,
{"slug": "compose-main-page-slug"},
{},
CONTENT_TYPE_PAGE_RESOURCE_CENTER,
{},
{
"slug": "compose-main-page-slug",
"slug": "vrc-main-page-slug",
"title": "",
},
{},
{
"slug": "vrc-main-page-slug",
"title": "",
"blurb": "",
},
),
(
"",
COMPOSE_MAIN_PAGE_TYPE,
{"slug": "compose-main-page-slug"},
CONTENT_TYPE_PAGE_RESOURCE_CENTER,
{},
{
"preview_title": "preview title",
"preview_blurb": "preview blurb",
"slug": "vrc-main-page-slug",
"title": "",
},
{},
{
"slug": "compose-main-page-slug",
"slug": "vrc-main-page-slug",
"title": "preview title",
"blurb": "preview blurb",
},
),
(
"",
COMPOSE_MAIN_PAGE_TYPE,
{"slug": "compose-main-page-slug"},
CONTENT_TYPE_PAGE_RESOURCE_CENTER,
{},
{
"preview_title": "preview title",
"preview_blurb": "preview blurb",
"slug": "vrc-main-page-slug",
"title": "",
},
{"description": "seo description"},
{
"slug": "compose-main-page-slug",
"slug": "vrc-main-page-slug",
"title": "preview title",
"blurb": "seo description",
},
),
(
"page title",
CONTENT_TYPE_CONNECT_HOMEPAGE,
{},
{
"slug": "homepage-slug",
"slug": "homepage-slug", # This will be ignored
},
{
"preview_title": "preview title",
"preview_blurb": "preview blurb",
},
{}, # SEO fields not present for non-Compose pages
{
"slug": "homepage-slug",
"slug": "home", # ie, there is no way to set the slug using Connect:Homepage
"title": "preview title",
"blurb": "preview blurb",
},
),
(
"page title",
CONTENT_TYPE_CONNECT_HOMEPAGE,
{},
{
# no slug field, so will fall back to default of 'home'
},
{
"preview_title": "preview title",
"preview_blurb": "preview blurb",
},
@ -962,26 +967,21 @@ def test_ContentfulPage__get_info_data__locale(
"compose page with slug, no title, no blurb",
"compose page with slug, title from entry, blurb from entry",
"compose page with slug, no title, blurb from seo",
"Non-Compose page with slug, title, blurb from entry",
"Non-Compose page with title, blurb from entry + PROOF SLUG IS NOT SET",
"Non-Compose page with default slug, title, blurb from entry",
],
)
def test_ContentfulPage__get_info_data__slug_title_blurb(
basic_contentful_page,
page_title,
page_type,
page_fields,
legacy_connect_page_fields,
entry_fields,
seo_fields,
expected,
):
basic_contentful_page.page = Mock()
basic_contentful_page.page.content_type.id = page_type
basic_contentful_page.page.fields = Mock(return_value=page_fields)
if page_title:
basic_contentful_page.page.title = page_title
else:
basic_contentful_page.page.title = ""
basic_contentful_page.page.fields = Mock(return_value=legacy_connect_page_fields)
assert (
basic_contentful_page._get_info_data__slug_title_blurb(
@ -1049,6 +1049,7 @@ def test_ContentfulPage__get_info_data__category_tags_classification(
{
"dummy": "seo fields",
"preview_image": "https://example.com/test-seo.png",
"description": "Test SEO description comes through",
},
{
"title": "test title",
@ -1068,6 +1069,7 @@ def test_ContentfulPage__get_info_data__category_tags_classification(
"seo": {
"dummy": "seo fields",
"image": "https://example.com/test-seo.png",
"description": "Test SEO description comes through",
},
},
),
@ -1169,11 +1171,11 @@ def test_ContentfulPage__get_info_data(
{
"dummy": "seo fields",
"preview_image": "https://example.com/test-seo.png",
"description": "Test SEO description comes through",
},
)
in mock__get_preview_image_from_fields.call_args_list
)
else:
assert mock__get_preview_image_from_fields.call_count == 1
mock__get_preview_image_from_fields.assert_called_once_with(entry_obj__fields)

84
bedrock/contentful/tests/test_contentful_commands.py
Просмотреть файл

@ -19,8 +19,8 @@ from bedrock.contentful.constants import (
ACTION_SAVE,
ACTION_UNARCHIVE,
ACTION_UNPUBLISH,
COMPOSE_MAIN_PAGE_TYPE,
CONTENT_TYPE_CONNECT_HOMEPAGE,
CONTENT_TYPE_PAGE_RESOURCE_CENTER,
)
from bedrock.contentful.management.commands.update_contentful import (
MAX_MESSAGES_PER_QUEUE_POLL,
@ -322,7 +322,7 @@ def test_update_contentful__queue_has_viable_messages__no_viable_message_found__
message_actions_sequence,
command_instance,
):
# Create is the only message that will not trigger a contenful poll in Dev
# Create is the only message that will not trigger a Contentful poll in Dev
assert settings.APP_NAME == "bedrock-dev"
messages_for_queue = _build_mock_messages(message_actions_sequence)
mock_sqs, mock_queue = _establish_mock_queue(messages_for_queue)
@ -616,9 +616,9 @@ def test_update_contentful__get_content_to_sync(
3,
["en-US"],
[
(COMPOSE_MAIN_PAGE_TYPE, "entry_1", "en-US"),
(COMPOSE_MAIN_PAGE_TYPE, "entry_2", "en-US"),
(COMPOSE_MAIN_PAGE_TYPE, "entry_3", "en-US"),
(CONTENT_TYPE_PAGE_RESOURCE_CENTER, "entry_1", "en-US"),
(CONTENT_TYPE_PAGE_RESOURCE_CENTER, "entry_2", "en-US"),
(CONTENT_TYPE_PAGE_RESOURCE_CENTER, "entry_3", "en-US"),
],
0,
),
@ -626,8 +626,8 @@ def test_update_contentful__get_content_to_sync(
3,
["en-US"],
[
(COMPOSE_MAIN_PAGE_TYPE, "entry_1", "en-US"),
(COMPOSE_MAIN_PAGE_TYPE, "entry_2", "en-US"),
(CONTENT_TYPE_PAGE_RESOURCE_CENTER, "entry_1", "en-US"),
(CONTENT_TYPE_PAGE_RESOURCE_CENTER, "entry_2", "en-US"),
],
1,
),
@ -635,9 +635,9 @@ def test_update_contentful__get_content_to_sync(
5,
["en-US"],
[
(COMPOSE_MAIN_PAGE_TYPE, "entry_2", "en-US"),
(COMPOSE_MAIN_PAGE_TYPE, "entry_3", "en-US"),
(COMPOSE_MAIN_PAGE_TYPE, "entry_4", "en-US"),
(CONTENT_TYPE_PAGE_RESOURCE_CENTER, "entry_2", "en-US"),
(CONTENT_TYPE_PAGE_RESOURCE_CENTER, "entry_3", "en-US"),
(CONTENT_TYPE_PAGE_RESOURCE_CENTER, "entry_4", "en-US"),
],
2,
),
@ -651,18 +651,18 @@ def test_update_contentful__get_content_to_sync(
3,
["en-US", "de", "fr", "it"],
[
(COMPOSE_MAIN_PAGE_TYPE, "entry_1", "en-US"),
# (COMPOSE_MAIN_PAGE_TYPE, "entry_1", "de"), # simulating deletion/absence from sync
(COMPOSE_MAIN_PAGE_TYPE, "entry_1", "fr"),
# (COMPOSE_MAIN_PAGE_TYPE, "entry_1", "it"),
# (COMPOSE_MAIN_PAGE_TYPE, "entry_2", "en-US"),
(COMPOSE_MAIN_PAGE_TYPE, "entry_2", "de"),
(COMPOSE_MAIN_PAGE_TYPE, "entry_2", "fr"),
(COMPOSE_MAIN_PAGE_TYPE, "entry_2", "it"),
(COMPOSE_MAIN_PAGE_TYPE, "entry_3", "en-US"),
# (COMPOSE_MAIN_PAGE_TYPE, "entry_3", "de"),
# (COMPOSE_MAIN_PAGE_TYPE, "entry_3", "fr"),
# (COMPOSE_MAIN_PAGE_TYPE, "entry_3", "it"),
(CONTENT_TYPE_PAGE_RESOURCE_CENTER, "entry_1", "en-US"),
# (CONTENT_TYPE_PAGE_RESOURCE_CENTER, "entry_1", "de"), # simulating deletion/absence from sync
(CONTENT_TYPE_PAGE_RESOURCE_CENTER, "entry_1", "fr"),
# (CONTENT_TYPE_PAGE_RESOURCE_CENTER, "entry_1", "it"),
# (CONTENT_TYPE_PAGE_RESOURCE_CENTER, "entry_2", "en-US"),
(CONTENT_TYPE_PAGE_RESOURCE_CENTER, "entry_2", "de"),
(CONTENT_TYPE_PAGE_RESOURCE_CENTER, "entry_2", "fr"),
(CONTENT_TYPE_PAGE_RESOURCE_CENTER, "entry_2", "it"),
(CONTENT_TYPE_PAGE_RESOURCE_CENTER, "entry_3", "en-US"),
# (CONTENT_TYPE_PAGE_RESOURCE_CENTER, "entry_3", "de"),
# (CONTENT_TYPE_PAGE_RESOURCE_CENTER, "entry_3", "fr"),
# (CONTENT_TYPE_PAGE_RESOURCE_CENTER, "entry_3", "it"),
],
6,
),
@ -685,7 +685,7 @@ def test_update_contentful__detect_and_delete_absent_entries(
for locale in locales_to_use:
for idx in range(total_to_create_per_locale):
ContentfulEntry.objects.create(
content_type=COMPOSE_MAIN_PAGE_TYPE,
content_type=CONTENT_TYPE_PAGE_RESOURCE_CENTER,
contentful_id=f"entry_{idx+1}",
locale=locale,
)
@ -713,7 +713,7 @@ def test_update_contentful__detect_and_delete_absent_entries__homepage_involved(
for locale in ["en-US", "fr", "it"]:
for idx in range(3):
ContentfulEntry.objects.create(
content_type=COMPOSE_MAIN_PAGE_TYPE,
content_type=CONTENT_TYPE_PAGE_RESOURCE_CENTER,
contentful_id=f"entry_{idx+1}",
locale=locale,
)
@ -722,27 +722,27 @@ def test_update_contentful__detect_and_delete_absent_entries__homepage_involved(
entries_processed_in_sync = [
(CONTENT_TYPE_CONNECT_HOMEPAGE, "home_1", "en-US"),
# (CONTENT_TYPE_CONNECT_HOMEPAGE, "home_2", "en-US"),
(COMPOSE_MAIN_PAGE_TYPE, "entry_1", "en-US"),
(COMPOSE_MAIN_PAGE_TYPE, "entry_1", "fr"),
(COMPOSE_MAIN_PAGE_TYPE, "entry_1", "it"),
# (COMPOSE_MAIN_PAGE_TYPE, "entry_2", "en-US"),
(COMPOSE_MAIN_PAGE_TYPE, "entry_2", "fr"),
# (COMPOSE_MAIN_PAGE_TYPE, "entry_2", "it"),
(COMPOSE_MAIN_PAGE_TYPE, "entry_3", "en-US"),
# (COMPOSE_MAIN_PAGE_TYPE, "entry_3", "fr"),
(COMPOSE_MAIN_PAGE_TYPE, "entry_3", "it"),
(CONTENT_TYPE_PAGE_RESOURCE_CENTER, "entry_1", "en-US"),
(CONTENT_TYPE_PAGE_RESOURCE_CENTER, "entry_1", "fr"),
(CONTENT_TYPE_PAGE_RESOURCE_CENTER, "entry_1", "it"),
# (CONTENT_TYPE_PAGE_RESOURCE_CENTER, "entry_2", "en-US"),
(CONTENT_TYPE_PAGE_RESOURCE_CENTER, "entry_2", "fr"),
# (CONTENT_TYPE_PAGE_RESOURCE_CENTER, "entry_2", "it"),
(CONTENT_TYPE_PAGE_RESOURCE_CENTER, "entry_3", "en-US"),
# (CONTENT_TYPE_PAGE_RESOURCE_CENTER, "entry_3", "fr"),
(CONTENT_TYPE_PAGE_RESOURCE_CENTER, "entry_3", "it"),
]
retval = command_instance._detect_and_delete_absent_entries(entries_processed_in_sync)
assert retval == 4
for ctype, contentful_id, locale in [
(CONTENT_TYPE_CONNECT_HOMEPAGE, "home_1", "en-US"),
(COMPOSE_MAIN_PAGE_TYPE, "entry_1", "en-US"),
(COMPOSE_MAIN_PAGE_TYPE, "entry_1", "fr"),
(COMPOSE_MAIN_PAGE_TYPE, "entry_1", "it"),
(COMPOSE_MAIN_PAGE_TYPE, "entry_2", "fr"),
(COMPOSE_MAIN_PAGE_TYPE, "entry_3", "en-US"),
(COMPOSE_MAIN_PAGE_TYPE, "entry_3", "it"),
(CONTENT_TYPE_PAGE_RESOURCE_CENTER, "entry_1", "en-US"),
(CONTENT_TYPE_PAGE_RESOURCE_CENTER, "entry_1", "fr"),
(CONTENT_TYPE_PAGE_RESOURCE_CENTER, "entry_1", "it"),
(CONTENT_TYPE_PAGE_RESOURCE_CENTER, "entry_2", "fr"),
(CONTENT_TYPE_PAGE_RESOURCE_CENTER, "entry_3", "en-US"),
(CONTENT_TYPE_PAGE_RESOURCE_CENTER, "entry_3", "it"),
]:
assert ContentfulEntry.objects.get(
content_type=ctype,
@ -752,9 +752,9 @@ def test_update_contentful__detect_and_delete_absent_entries__homepage_involved(
for ctype, contentful_id, locale in [
(CONTENT_TYPE_CONNECT_HOMEPAGE, "home_2", "en-US"),
(COMPOSE_MAIN_PAGE_TYPE, "entry_2", "en-US"),
(COMPOSE_MAIN_PAGE_TYPE, "entry_2", "it"),
(COMPOSE_MAIN_PAGE_TYPE, "entry_3", "fr"),
(CONTENT_TYPE_PAGE_RESOURCE_CENTER, "entry_2", "en-US"),
(CONTENT_TYPE_PAGE_RESOURCE_CENTER, "entry_2", "it"),
(CONTENT_TYPE_PAGE_RESOURCE_CENTER, "entry_3", "fr"),
]:
assert not ContentfulEntry.objects.filter(
content_type=ctype,

2
bedrock/firefox/urls.py
Просмотреть файл

@ -230,4 +230,4 @@ urlpatterns = (
# Contentful
if settings.DEV:
urlpatterns += (path("firefox/more/<content_id>/", views.FirefoxContenful.as_view()),)
urlpatterns += (path("firefox/more/<content_id>/", views.FirefoxContentful.as_view()),)

2
bedrock/firefox/views.py
Просмотреть файл

@ -1072,7 +1072,7 @@ class FirefoxMobileView(L10nTemplateView):
return [template]
class FirefoxContenful(L10nTemplateView):
class FirefoxContentful(L10nTemplateView):
def get_context_data(self, **kwargs):
ctx = super().get_context_data(**kwargs)
content_id = ctx["content_id"]

1
contentful_migrations/.env.dist Normal file
Просмотреть файл

@ -0,0 +1 @@
CONTENTFUL_ACCESS_TOKEN=SETME

3
contentful_migrations/.gitignore поставляемый Normal file
Просмотреть файл

@ -0,0 +1,3 @@
node_modules
.env
*.log

55
contentful_migrations/README.md Normal file
Просмотреть файл

@ -0,0 +1,55 @@
# Contentful Migration Scripts
## What is this?
This directory contain scripts (written in Javascript) that execute Contentful migrations related to content used in Bedrock - eg, changes to the schema or data of a particular Contentful environment.
It does not, currently, contain a complete CMS-as-Code approach. Each migration in here is a
one-off scripted change intended for one-off use, outside of CI. The migrations are run via
the [contentful-migrations](https://github.com/jungvonmatt/contentful-migrations) framework,
which wraps Contentful's own `contentful-migration` library.
## How do I use it?
**Groundwork**
* Have Node.js installed (any recent/LTS version will be ok)
* `$ cd /path/to/checkout/of/bedrock/contentful_migrations/`
* `$ npm install` to add the `contentful-migrations` tool
* `$ npx contentful login` to get an authentication token via your browser
### To apply an existing migration
`npx migrations execute -e TARGET_CONTENTFUL_ENVIRONMENT -v migrations/MIGRATION_FILE_NAME.cjs`
This will run the migration and also set a `Migrations` Entry in Contentful, with the number of
the migration and a success state. This record is useful in showing us what migrations have
run on what environment.
### To unmark a migration as applied
Unlike Django Migrations, you can re-apply a Contentful migration without having to first roll
it back, as long as the operations in that migration still make sense compared to the state of
your target environment. However, say you manually un-do the changes set up by a migration and
you want to show others that it effectively no longer is applied there. In that case, do this:
`npx migrations version -e TARGET_CONTENTFUL_ENVIRONMENT -v --remove migrations/MIGRATION_FILE_NAME.cjs`
Honestly speaking, there's not a lot of value in this command right now, but maybe we can
contribute some improvements to make it more like Django Migrations in its locking/denying
behaviour.
### To make a new migration
`npx migrations generate` and then fill in the skeleton migration file created. The number used in the default filename is a timestamp, showing the order migration files were created in. You should
add to it so the filename helps indicate what the file does, too.
## Is there a safe mode or dry run option?
No. If you're unsure about what the script does, fork the production environment in Contentful to a new environment and then try the script on that first.
## Roadmap
* [DONE] - use migrations (via contentful-migrations framework) to move from Legacy Compose to new Compose. This also acts as a POC for using it within Bedrock. Migrations will be manually run for now.
* [TODO] - plan how we'll move to [CMS-as-Code](https://www.contentful.com/help/cms-as-code/) for Bedrock pages that use Contentful, including CI integration so that migrations are auto-run.
* [TODO] - Move to CMS-as-Code, formally.

64
contentful_migrations/migrations/1661197009323--new-compose--add-fields.cjs Normal file
Просмотреть файл

@ -0,0 +1,64 @@
/*
* This Source Code Form is subject to the terms of the Mozilla Public
* License, v. 2.0. If a copy of the MPL was not distributed with this
* file, You can obtain one at https://mozilla.org/MPL/2.0/.
*/
const { withHelpers } = require('@jungvonmatt/contentful-migrations');
/**
* Contentful migration
* API: https://github.com/contentful/contentful-migration
* Editor Interfaces: https://www.contentful.com/developers/docs/extensibility/app-framework/editor-interfaces/
*/
const RESOURCE_CENTER_PAGE = 'pagePageResourceCenter';
const LEGACY_COMPOSE_PAGE = 'page';
module.exports = withHelpers(async function (migration, context, helpers) {
const pagePageResourceCenterMigration =
migration.editContentType(RESOURCE_CENTER_PAGE);
// 1. Add fields to our custom page that were previously on the Compose: Page model
pagePageResourceCenterMigration
.createField('title')
.name('Page title')
.type('Symbol')
.localized(true)
.required(false) // change this to True later on, after population
.validations([])
.disabled(false)
.omitted(false);
pagePageResourceCenterMigration
.createField('slug')
.name('Slug')
.type('Symbol')
.localized(true)
.required(false) // change this to True later on, after population
.validations([])
.disabled(false)
.omitted(false);
pagePageResourceCenterMigration
.createField('seo')
.name('SEO metadata')
.type('Link')
.linkType('Entry')
.localized(true)
.required(false)
.validations([])
.disabled(false)
.omitted(false);
// 2. Add the “Aggregate Root” annotation to the page type so it can work
// as a page without needing Compose:Page to parent it
pagePageResourceCenterMigration.setAnnotations([
'Contentful:AggregateRoot'
]);
// 3. Activate the page type
// DISABLED because we get a HTTP 409: Conflict, but with no details, so am
// assuming it's because the page is _already_ live. Will see...
// const resp = await context.makeRequest({
// method: 'PUT',
// url: `/content_types/${RESOURCE_CENTER_PAGE}/published`
// });
});

104
contentful_migrations/migrations/1662471934889-new-compose--populate-data.cjs Normal file
Просмотреть файл

@ -0,0 +1,104 @@
/* eslint-env node */
const { withHelpers } = require('@jungvonmatt/contentful-migrations');
// DEPENDS on 1661197009323--new-compose--add-fields.cjs
/**
* Contentful migration
* API: https://github.com/contentful/contentful-migration
* Editor Interfaces: https://www.contentful.com/developers/docs/extensibility/app-framework/editor-interfaces/
*/
module.exports = withHelpers(async function (migration, context, helpers) {
// Copy field data from Compose:Page parent to now-standalone child
const RESOURCE_CENTER_PAGE = 'pagePageResourceCenter';
const LEGACY_COMPOSE_PAGE = 'page';
migration.transformEntries({
contentType: LEGACY_COMPOSE_PAGE,
from: ['content', 'seo', 'title', 'slug'],
to: ['content', 'seo', 'title', 'slug'],
transformEntryForLocale: async (fromFields, currentLocale) => {
if (currentLocale != 'en-US') {
return;
}
console.log(
'IMPORTANT: handling en-US locale ONLY and skipping all other locales'
);
// Moving data across entries doesn't appear supported via helpers, so
// let's patch the relevant pagePageResourceCenter Entry
// via the API manually:
// https://www.contentful.com/developers/docs/references/content-management-api/#/reference/entries/entry/patch-an-entry/console/js
if (
!fromFields['content'][currentLocale].sys ||
!fromFields['seo'][currentLocale].sys
) {
console.log(
'Missing data, so skipping entry for ',
currentLocale
);
return;
}
const contentPageEntryId =
fromFields['content'][currentLocale].sys.id;
const seoObjectEntryId = fromFields['seo'][currentLocale].sys.id;
// Get the actual content entry to get the data we need to fill in,
// so we can check if it's a pagePageResourceCenter
const contentPageEntry = await context.makeRequest({
method: 'get',
url: `/entries/${contentPageEntryId}`
});
if (
contentPageEntry.sys.contentType.sys.id == RESOURCE_CENTER_PAGE
) {
const payload = [
{
op: 'add',
path: '/fields/title',
value: {
[currentLocale]: fromFields['title'][currentLocale]
}
},
{
op: 'add',
path: '/fields/slug',
value: {
[currentLocale]: fromFields['slug'][currentLocale]
}
},
{
op: 'add',
path: '/fields/seo',
value: {
[currentLocale]: {
sys: {
type: 'Link',
linkType: 'Entry',
id: seoObjectEntryId
}
}
}
}
];
result = await context.makeRequest({
method: 'patch',
url: `/entries/${contentPageEntryId}`,
headers: {
'Content-Type': 'application/json-patch+json',
'X-Contentful-Version': contentPageEntry.sys.version
},
data: payload
});
console.log('Result of data migration =>', result);
}
// Returning no data signals no change to the Compose:Page (from where we get fromFields),
// but we'll have executed the Contentful Management API calls to update data in
// the target page anyway.
return;
}
});
});

72
contentful_migrations/migrations/1662646314014-new-compose--drop-old-compose-page-data-and-content-type.cjs Normal file
Просмотреть файл

@ -0,0 +1,72 @@
/* eslint-env node */
const { withHelpers } = require('@jungvonmatt/contentful-migrations');
// DEPENDS on 1661197009323--new-compose--add-fields.cjs and its prerequisite migration
/**
* Contentful migration
* API: https://github.com/contentful/contentful-migration
* Editor Interfaces: https://www.contentful.com/developers/docs/extensibility/app-framework/editor-interfaces/
*/
const sleep = (milliseconds) =>
new Promise((res) => setTimeout(res, milliseconds));
module.exports = withHelpers(async function (migration, context, helpers) {
const LEGACY_COMPOSE_PAGE = 'page';
// // 0. Get all Compose:Page entries
allEntries = await context.makeRequest({
method: 'get',
url: '/entries'
});
const legacyComposePageEntries = allEntries.items.filter(
(entry) => entry.sys.contentType.sys.id == LEGACY_COMPOSE_PAGE
);
// 1. Unpublish all Compose:Page entries
await legacyComposePageEntries.forEach((composePageEntry) => {
if (composePageEntry.sys.publishedVersion) {
console.log(`Unpublishing Compose:Page ${composePageEntry.sys.id}`);
const resp = context.makeRequest({
method: 'delete',
url: `/entries/${composePageEntry.sys.id}/published`,
headers: {
'X-Contentful-Version':
composePageEntry.sys.publishedVersion
}
});
console.log('Compose:Page unpublishing response:', resp);
} else {
console.log(
`Compose:Page ${composePageEntry.sys.id} was not Published, so skipping`
);
}
});
// Give Contentful a moment to breathe before moving on, to avoid race
// conditions and/or rate limiting
console.log('Sleeping for 2s...');
await sleep(2000);
// 2. Delete all Compose:Page entries
await legacyComposePageEntries.forEach((composePageEntry) => {
console.log(`Deleting Compose:Page ${composePageEntry.sys.id}`);
const resp = context.makeRequest({
method: 'delete',
url: `/entries/${composePageEntry.sys.id}`,
headers: {
'X-Contentful-Version': composePageEntry.sys.version
}
});
console.log('Compose:Page deletion response:', resp);
});
// Give Contentful another moment to breathe before moving on
console.log('Sleeping for another 2s...');
await sleep(2000);
// 3. Delete Compose:Page ContentType
result = await migration.deleteContentType(LEGACY_COMPOSE_PAGE);
console.log('Deleted Compose:Page content type. Result:', result);
});

19
contentful_migrations/migrations/1662751123660-new-compose--update-ui-fields.cjs Normal file
Просмотреть файл

@ -0,0 +1,19 @@
/* eslint-env node */
const { withHelpers } = require('@jungvonmatt/contentful-migrations');
// Improve the UI layout to help the developer experience.
/**
* Contentful migration
* API: https://github.com/contentful/contentful-migration
* Editor Interfaces: https://www.contentful.com/developers/docs/extensibility/app-framework/editor-interfaces/
*/
const RESOURCE_CENTER_PAGE = 'pagePageResourceCenter';
module.exports = withHelpers(async function (migration, context, helpers) {
const resourceCenterPage = migration.editContentType(RESOURCE_CENTER_PAGE);
resourceCenterPage.moveField('title').afterField('name');
resourceCenterPage.moveField('slug').afterField('title');
resourceCenterPage.moveField('seo').afterField('slug');
});

16
contentful_migrations/migrations/1663242603331-legacy-compose--make-new-title-and-slug-fields-required.cjs Normal file
Просмотреть файл

@ -0,0 +1,16 @@
/* eslint-env node */
const { withHelpers } = require('@jungvonmatt/contentful-migrations');
/**
* Contentful migration
* API: https://github.com/contentful/contentful-migration
* Editor Interfaces: https://www.contentful.com/developers/docs/extensibility/app-framework/editor-interfaces/
*/
const RESOURCE_CENTER_PAGE = 'pagePageResourceCenter';
module.exports = withHelpers(async function (migration, context, helpers) {
const resourceCenterPage = migration.editContentType(RESOURCE_CENTER_PAGE);
resourceCenterPage.editField('title').required(true);
resourceCenterPage.editField('slug').required(true);
});

39
contentful_migrations/migrations/1663245283845-new-compose--publish-all-vrc-pages.cjs Normal file
Просмотреть файл

@ -0,0 +1,39 @@
/* eslint-env node */
const { withHelpers } = require('@jungvonmatt/contentful-migrations');
/**
* Contentful migration
* API: https://github.com/contentful/contentful-migration
* Editor Interfaces: https://www.contentful.com/developers/docs/extensibility/app-framework/editor-interfaces/
*/
// Publish all Resource Center pages
const RESOURCE_CENTER_PAGE = 'pagePageResourceCenter';
module.exports = withHelpers(async function (migration, context, helpers) {
allEntries = await context.makeRequest({
method: 'get',
url: '/entries'
});
const resourceCenterPageEntries = allEntries.items.filter(
(entry) => entry.sys.contentType.sys.id == RESOURCE_CENTER_PAGE
);
await resourceCenterPageEntries.forEach((rcPageEntry) => {
if (rcPageEntry.sys.publishedVersion < rcPageEntry.sys.version) {
console.log(`Publishing changed page ${rcPageEntry.sys.id}`);
const resp = context.makeRequest({
method: 'put',
url: `/entries/${rcPageEntry.sys.id}/published`,
headers: {
'X-Contentful-Version': rcPageEntry.sys.version
}
});
} else {
console.log(
`Page ${rcPageEntry.sys.id} was already Published, so skipping`
);
}
});
});

3956
contentful_migrations/package-lock.json сгенерированный Normal file
Просмотреть файл

Разница между файлами не показана из-за своего большого размера Загрузить разницу

19
contentful_migrations/package.json Normal file
Просмотреть файл

@ -0,0 +1,19 @@
{
"name": "bedrock-contentful-migration-scripts",
"version": "0.0.1",
"type": "module",
"description": "Migration scripts to update schema or data in a specific Contentful space and environment",
"scripts": {
"test": "echo \"Error: no test specified\" && exit 1"
},
"author": "",
"license": "MPL-2.0",
"dependencies": {
"@jungvonmatt/contentful-migrations": "^5.3.2"
},
"migrations": {
"storage": "content",
"migrationContentTypeId": "contentful-migrations",
"directory": "migrations"
}
}

39
docs/architecture/decisions/0009-use-manage-contentful-migrations.md Normal file
Просмотреть файл

@ -0,0 +1,39 @@
# 9. Manage Contentful schema state via migrations
Date: 2022-09-09
## Status
Accepted
## Context
Our chosen CMS Contentful is powerful and can be configured via its UI quite
easily. However, wanted to bring this under control using migrations so that
changes are explicit, reviewable, repeatable and stored. This would be a key
part of moving to a "CMS-as-Code" approach to using Contentful, where
content-type changes and data migrations (outside of regular content entry)
are managed via code.
## Decision
We wanted to have as close as possible to the experience provided by the
excellent Django Migrations framework, where we would:
* be able to script migrations, rather than resort to "clickops"
* be able to apply them individually or en masse
* be able to store the state of which migrations have/have not been applied
in a central datastore (and ideally Contentful)
We experimented with hand-cutting our own framework, which was looking viable,
but then we came across https://github.com/jungvonmatt/contentful-migrations
which does all of the above. We've evaluated it and it seems fit for purpose,
even if it has some gaps, so we've adopted it as our current way to manage and
apply migrations to our Contentful setup.
## Consequences
We've gained a tool that enables code-based changes to Contentful, which helps
in two ways:
1) It enables and eases the initial work to migrate from Legacy Compose to
new Compose (these are both ways of structuring pages in Contentful)
2) It lays tracks for moving to CMS-as-Code