* Add django-rq to the project, with initial configuration for two redis-backed queues if redis is available
* Add Django-rq to project, including link from (now necessary) Django admin's userlinks
* Update Docker support to run a local worker node for RQ
* Add Sentry integration for Redis and for RQ
* Add a make command to run the worker queue
Temporarily loads env vars from .env, looking for REDIS_URL
* Make a new setting that makes it unambiguous whether a task queue is available
* Add helper that will enqueue a task or immediately execute it, depending on whether the task queue is available
Note that if it finds a queue is available it will put a job in there, regardless of whether the queue is being processed by a worker
* Switch image-rendition pre-generation to use the task queue, if available
* Add initial, basic, docs about the task queue
* Improve logging around task queue usage
* Fix formatting of Makefile
Thanks @janbrasna!
* Typo fixes in documentation
Thanks @janbrasna!
* Add Wagtail 5.2 LTS as a dependency
* Drop in wagtail-localize, ready for i18n/l10n
Also fix up some test regressions as a result of this change, including
changing how Careers renders a custom 404-page message
* Support detection of CMS-based pages so that we don't falsely redirect away from pages managed by wagtail-localize
* Ensure that pages fall back (in their enriety) if we don't have a page with the matching lang code in the CMS
* Back out hack to 404.html for careers-related 404 -- unnecessary following i18n/routing refactor
* Back out unnecessary follow=True on the 404 test
* Make database config dynamic, but default to current local sqlite DB
For most setups, without a DATABASE_URL set, nothing changes
* Add psycopg3 dependency, for postgres support
* Avoid hitting the productdetails table during bootup
Because this makes it impossible to boot Bedrock with an empty database
* Add postgres DB to docker-compose, initially for local-dev use only
* Support bootstrapping an empty postgres database with bedrock data
* Update models in security and contentful models' schemae to handle invalid
data that sqlite tolerated, but postgres does not
* Add a shell script that will bootstrap an empty postgres database with
data either from the main external sources, or from the sqlite DB that
bedrock downloads
* Add db-to-sqlite lib to project
Downgrade psycopg(3) to psycopg2 due to hard dependencies of db-to-sqlite
* Rename postgres-bootstrapping commmand to kebab-case for consistency
* Add script to export postgres DB to sqlite, redacting auth_user table along the way
* Expand DB export script to redact or skip certain Wagtail-related tables/columns
* Switch to using loaddata/dumpdata instead of db-to-sqlite, because we were having column-type drift that was breaking Wagtail
* Tune the dumpdata export a little
Based on https://github.com/wagtail/bakerydemo?tab=readme-ov-file#preparing-this-archive-for-distribution
* Rename postgres-filling command to be less ambiguous
* Include ProductRelease in dumped data, because now we can after fixing the NotesField in an earlier commit
* DRY up the success/exit code
* Revert unnecessary lazy-making of product_details
* make export-to-sqlite script more robust and clearer
* Add simple guard rails to ensure DATABASE_URL points to a postgres DB when these scripts are run
* Add test coverage for data-scrubber management command
(Originally written by Copilot but heavily refactored and made appropriate)
* Refactor Pocket URLConf ahead of making it a first-class URLconf
For now, the pages are still available via /externalpages/pocket/ but
the URLconf is no longer opinionated about having to have the /externalpages/
namespace in there. This means we can plug it in at the root level when
in Pocket-only mode
* Add support to package.json for easy use of Pocket mode or Mozorg mode
npm run in-mozorg-mode
npm run in-pocket-mode
* Support running in Mozorg or Pocket modes in dev or prod via Docker, via Makefile commands
* Support Mozorg or Pocket mode via env var
Setting the SITE_MODE env var to Mozorg or Pocket will now switch the site
into serving only the URLs for that particular mode.
When in Pocket mode, its pages are served from the root path, not from
the 'externalpages' namespace/module name
The default behaviour, for now, remains as is: all URLs will be served, and
Pocket URLs are served as 'external/pocket/*`
* Minor comment shuffle
* Flatten externalpages/pocket to just pocket
* Make Mozorg the default mode, with Pocket enabled via SITE_MODE env var
* Update docs to outline how to invoke Pocket mode
* Amend link to infringement reporting to be hard-coded, because can no longer reverse() it
* Amend link to bug reporting to be hard-coded, because can no longer reverse() it
* Do not use the redirection middleware when in Pocket mode - it clashes with some Pocket URLs
* Hard-code a careers listing URL, because can not reverse Mozorg URLs in Pocket mode
* Drop redunant Default SITE_MODE from docker config
* Disable Pocket integration tests, temporarily
* Ensure Pocket mode does not use bedrock.redirects as an INSTALLED_APP
* Remove pocket reference from mozorg robots.txt
* Add in robots.txt for Pocket mode, based on current state of https://getpocket.com/robots.txt
* Revert "Add in robots.txt for Pocket mode, based on current state of https://getpocket.com/robots.txt"
This reverts commit 5934b4d613.
* Remove straggling markup from diff resolution
* Move Bedrock to pip-compile-multi for easier Python dependency management
This changeset adds tooling to ease dependency management and also rationalises
our requirements files.
Before, we were just using hashin to manually hash pinned deps straight into a requirements file
Now we're using pip-compile-multi, which sits on top of pip-tools to do this.
We now get:
* Simpler syntax for adding and pinning dependencies via *.in files
* Automatic hash generation when the *.txt requirements files are produced
The dependency compilation/update tooling runs in a Docker container, so will be compatible
with the deployed service's containers.
We're also rationalising the existing split of dependency files:
* base -> being retired and used as the basis for prod requirements
* migration -> being retired and the two deps still useful to us (for moz-l10n-lint)
added to dev deps
* dev -> now extends from the prod requirements. We're not too concerned about image
size for dev and test builds
* prod -> still exists, but includes the base deps
* docs -> still exists as a standalone file, but also follows the "*.in"-file pattern
* Regenerate dependency files using pip-compile-multi
Note that to avoid clashes, the following balances were made:
* Keep meinheld at the lower version used in prod.txt, not the dev.txt one -- for now at least
* Downgrade Markdown to 3.3 to avoid a clash over importlib-metadata version
* Remove importlib-metadata==4.10.1 altogether as a hard pin and let pip-compile-multi work out the best fit
* Update docs to reference pip-compile-multi, replacing now-redundant notes on hashin
* Update Dockerfile to copy over and use freshly recut dependency files
* Attempting to tune deps to allow local builds to work, not just Docker ones
* Update pip-compile-multi config to inject a custom header that explains how to rebuild reqs
* Update Bedrock to use Python 3.9
* Update base images
* Update CI
* Update dependencies to make install run -- this involved manually using hashin to upgrade two hashed deps (greenlet and meinheld) then re-running make compile-requirements to update the top-level hash in prod.txt. It's a bit of a chicken-and-egg situation when the deps are built/re-locked in a container but you can't build the container itself unless the deps are viable, but it worked
* Upgrade everett in order to remove configobj, which is redundant and causing local install issues on MacOS M1
* Drop backports.cached-property and typed-ast from dev reqs because we don't need them on 3.9
* Update docs with local-installation guidance for pyenv and pyenv-virtualenv
* Remove 'upgrade requirements' option
Given that the --upgrade flag is implicitly / by-default true with
pip-compile-multi anyway, plus the fact we're hard-pininng everything,
there's no point having an explicit 'upgrade' path - so let's remove it
* Docs tweak to suggest simpler virtualenv name
* Update help option in Makefile
* Pin version of pip in the compile-requirements script
When unpinned, the build suddenly broke, so we're keeping it under strict
limits for now
* Upgrade Django to 2.2.27
* Upgrade newrelic package to latest, incl py3.9 support
* Switch to Python 3.9 Debian bullseye image, from buster
* Rationalise dependency input files to remove over-pinned subdeps
When we moved from hand-managed requirements.txt files, we were taking on files
that had literally every dependency and thier sub-deps in them. We don't want
the input (*.in) files to reference those subdeps, so this changeset tries to
thin things out and remove them
Note that the diff shows this was successful - there are very few changes to
the dependencies being mentioned in the output *.txt files, and the ones that
are there are all deliberate changes (eg removing 'pbr')
* Drop unused tenacity dep, bump APScheduler and link to a Python 3.9-patched version of mdx-outline
* Add --require-hashes option to pip usage in Dockerfile
It's implicitly set because the reqs files feature --hash=XXX
but better to be explicit
* Drop what appear to be redundant top-level dev dependencies
regex, pep8 and wcwidth appear to be subdeps that don't need pinning.
The others appear to not be in used, based on a search of the codebase.
Tests till pass
* Thin out some unnecessary top-level deps in prod.in
- funcsigs - old backport, redundant
- lxml - over-pinned subdep of BeautifulSoup?
- typing_extensions - over-pinned subdep
- zipp - over-pinned subdep
* Cap pip version to 21 for pip-compile-multi for now
* Reinstate lxml as a first-class dep: BeautifulSoup needs it as a user-specified parser
* Fix typo in pip-compile-multi header
* Hard-pin latest working combo of pip + pip-tools in compile-requirements.sh
* Update docs explaining why we're using 3.9.10 locally
These changes will allow the build to succeed on machines with
non-intel processor architectures. This is so far mostly an issue
for Mac computers with the M1 chip (ARM64 arch).
Have the app download the right data on startup. If that fails it won't
start the app and so the deployment will fail, but the existing
deployment will stay up.
* Fix dict cache default timeout
* Add fallback strings and checking for translations in template
* Add ftl alias and ftl_lazy for use in py files
* Move git repos and add fluent to l10n_update
* Add percent_translated and required_messages to the l10n object
* Add management commands for converting en-US .lang file and templates
* Add command to convert en-US .lang file to .ftl file
* Using data from above command add a command that will convert a
template to use the .ftl strings
* Update the l10n_update command to update both external l10n repos
* Convert the /mission/ page to Fluent
* Add command to convert .lang translations to .ftl
* Copy en .ftl file when porting translations
* Removes the original string hash comments as well.
* Add Fluent docs.
* Add indentation and key sorting to metadata JSON output
* Add FTL file linting command
* Update mission template for FTL and remove FTL specific template
* Set l10n cache to 10s when DEBUG=True
* Fix and add tests for get_l10n_path
* Fix some style mistakes in docs and code
* Support multiline group comments for required strings
fix#7725, fix#7822, and fix#7726
* Install Sphinx markdown deps
* Configure Sphinx with markdown support
* Switch to RTD Sphinx theme for local too
* Fix a bunch of Sphinx warnings for docs
* Separate docs deps into a requirements file for RTD to use
Most developers won't need to build their own Docker images. Pulling
our pre-built ones from the Docker Hub will work perfectly for them.
This patch changes the install instructions to reflect this as well
as changing the majority of the Makefile commands to depend on pull
rather than build.
* Move from django-pipeline to Gulp for static-assets
* Use Gulp tasks to compile less and sass
* Use Gulp tasks to concatinate files into bundles
* Use Gulp tasks to minify files for deployment
* Use Gulp tasks for development to watch for changes
* Use BrowserSync to serve development static files and refresh the
in-progress page
* Update Docker setup to use multi-stage build
* Update Makefile to build and run the docker setup
* Update docs to recommend Docker-based development
* Update deployment and testing to also use the Makefile
* Add rebuild of SASS bundles when library files change
* Add an intermediate build directory just for LESS and SASS
* Avoid issues with ambiguous imports when .css and .scss in the same
directory
* Set deployment docker image in git env var script
Rob Hudson
Steve Jalim
Alex Gibson
Paul McLanahan
Giorgos Logiotatidis
Paul McLanahan
Josh Mize