* Add support for animated GIF processing in Wagtail with Wand
* Stop sending the BedrockImage object instance into the task queue
Instead, just send its ID and then pluck the model from the DB as part of the task
In order to run the postgres-to-sqlite export script in k8s, we need
to have sqlite3 available there.
This changeset moves it from the 'devapp' image to the 'app-base' image,
making it also available in the 'release' image, which we use in production
* [Tiny] Improve error messaging if git repo interaction fails
* Override the Fluent repo-related config in Pocket mode
Note that for Pocket mode, we don't need an intermediary mozmeao/* repo to
hold the translations while we calculate activation metadata via a CI task
(which does happen for Mozorg). Why? All locales in Pocket should be 100%
ready to go, because they are translated by a vendor, whereas Mozorg has
community translation contributions as well, which aren't always exhaustive.
As a result, both FLUENT_REPO and FLUENT_L10N_TEAM_REPO point to the same repo
* Update Bedrock CI step to try to open a PR against the l10n repo
...if there are changes to the l10n files.
Note that we try to update both www-l10n (Mozorg) and pocket-www-l10n (Pocket)
each time it's run, with the plan that if it's run against an irrelevant repo
no changes will be detected, even though it's running twice
(This may need tweaking once it's running in CI properly, of course)
* Update settings.get_dev_languages to take params, to make re-use elsewhere easier
* Initial pass at setting up DEV and PROD languages for Pocket mode
* Ensure Pocket always uses all available production langs
All locales in Pocket should be 100% ready to go, because they are
translated by a vendor, whereas Mozorg has community translation
contributions as well, which aren't always exhaustive.
* Wind back "Update settings.get_dev_languages to take params, to make re-use elsewhere easier"
This reverts some of the work done in c653640ae56cf21e961c461a421d6dd5fa54efcd.
* Make DEV_LANGUAGES the same as PROD_LANGUAGES as there's no need not to
* Ensure Pocket-related l10n files are copied into Dockerimage during build
* Make the l10n_update command pull from both Mozorg and Pocket Fluent repos
* Split out pocket-l10n PR step in CI, so that Mozorg and Pocket steps run separately
* Update LANGUAGE_CODE for Pocket mode to be just 'en'
* Update output messaging to help keep track of FTL updates
* Nitfix: make env setting in Docker consistent
* Tidy up l10n-related settings following code review
* Nit: updating comments
* Fixup: Add quotes around bash variable substitution
* Tweak locale fallbacks
* Update locale setup for Spanish, but more work needed
Just noting some complexity to unpick here. getpocket.com/es-la/ exists, and
so does getpocket.com/es/. However it is sounding like /es/ needs to use
`es-ES` content from the vendor and /es-la/ needs to use `es` from the vendor.
Also note that Pocket currently uses `/es-la/` not `/es-LA/`
This all may require some extra wiring, as it feels like it's flipping over
the behaviour we have in bedrock.
* Move non-official es-la locale to FALLBACK_LOCALES
* Drop es-MX, es-CL, es-AR from FALLBACK_LOCALES - unnecessary
If the language_REGION locale string doesn't map to a specific locale, we try
to map the two-letter locale, so we don't need the variant locales for es.
* Drop unnecessary inclusion of en-* locale variants -- they'll fall back to /en/ fine without config
* Adding cdn tests
Using a cdn mark to allow skipping or only running the cdn tests
uses ssllabs to get the tls/cipher information
That can be fairly slow, so trying to ensure it only happens once per run - so caching to a json file
* migrating existing tests into the functional area
* formatting failed in circle
* Move Bedrock to pip-compile-multi for easier Python dependency management
This changeset adds tooling to ease dependency management and also rationalises
our requirements files.
Before, we were just using hashin to manually hash pinned deps straight into a requirements file
Now we're using pip-compile-multi, which sits on top of pip-tools to do this.
We now get:
* Simpler syntax for adding and pinning dependencies via *.in files
* Automatic hash generation when the *.txt requirements files are produced
The dependency compilation/update tooling runs in a Docker container, so will be compatible
with the deployed service's containers.
We're also rationalising the existing split of dependency files:
* base -> being retired and used as the basis for prod requirements
* migration -> being retired and the two deps still useful to us (for moz-l10n-lint)
added to dev deps
* dev -> now extends from the prod requirements. We're not too concerned about image
size for dev and test builds
* prod -> still exists, but includes the base deps
* docs -> still exists as a standalone file, but also follows the "*.in"-file pattern
* Regenerate dependency files using pip-compile-multi
Note that to avoid clashes, the following balances were made:
* Keep meinheld at the lower version used in prod.txt, not the dev.txt one -- for now at least
* Downgrade Markdown to 3.3 to avoid a clash over importlib-metadata version
* Remove importlib-metadata==4.10.1 altogether as a hard pin and let pip-compile-multi work out the best fit
* Update docs to reference pip-compile-multi, replacing now-redundant notes on hashin
* Update Dockerfile to copy over and use freshly recut dependency files
* Attempting to tune deps to allow local builds to work, not just Docker ones
* Update pip-compile-multi config to inject a custom header that explains how to rebuild reqs
* Update Bedrock to use Python 3.9
* Update base images
* Update CI
* Update dependencies to make install run -- this involved manually using hashin to upgrade two hashed deps (greenlet and meinheld) then re-running make compile-requirements to update the top-level hash in prod.txt. It's a bit of a chicken-and-egg situation when the deps are built/re-locked in a container but you can't build the container itself unless the deps are viable, but it worked
* Upgrade everett in order to remove configobj, which is redundant and causing local install issues on MacOS M1
* Drop backports.cached-property and typed-ast from dev reqs because we don't need them on 3.9
* Update docs with local-installation guidance for pyenv and pyenv-virtualenv
* Remove 'upgrade requirements' option
Given that the --upgrade flag is implicitly / by-default true with
pip-compile-multi anyway, plus the fact we're hard-pininng everything,
there's no point having an explicit 'upgrade' path - so let's remove it
* Docs tweak to suggest simpler virtualenv name
* Update help option in Makefile
* Pin version of pip in the compile-requirements script
When unpinned, the build suddenly broke, so we're keeping it under strict
limits for now
* Upgrade Django to 2.2.27
* Upgrade newrelic package to latest, incl py3.9 support
* Switch to Python 3.9 Debian bullseye image, from buster
* Rationalise dependency input files to remove over-pinned subdeps
When we moved from hand-managed requirements.txt files, we were taking on files
that had literally every dependency and thier sub-deps in them. We don't want
the input (*.in) files to reference those subdeps, so this changeset tries to
thin things out and remove them
Note that the diff shows this was successful - there are very few changes to
the dependencies being mentioned in the output *.txt files, and the ones that
are there are all deliberate changes (eg removing 'pbr')
* Drop unused tenacity dep, bump APScheduler and link to a Python 3.9-patched version of mdx-outline
* Add --require-hashes option to pip usage in Dockerfile
It's implicitly set because the reqs files feature --hash=XXX
but better to be explicit
* Drop what appear to be redundant top-level dev dependencies
regex, pep8 and wcwidth appear to be subdeps that don't need pinning.
The others appear to not be in used, based on a search of the codebase.
Tests till pass
* Thin out some unnecessary top-level deps in prod.in
- funcsigs - old backport, redundant
- lxml - over-pinned subdep of BeautifulSoup?
- typing_extensions - over-pinned subdep
- zipp - over-pinned subdep
* Cap pip version to 21 for pip-compile-multi for now
* Reinstate lxml as a first-class dep: BeautifulSoup needs it as a user-specified parser
* Fix typo in pip-compile-multi header
* Hard-pin latest working combo of pip + pip-tools in compile-requirements.sh
* Update docs explaining why we're using 3.9.10 locally
Before, bpython would complain it couldn't write to /home/webdev/.pythonhist because it and the parent dir did not exist. Adding both with the appropriate permissions seems the smallest change to make the history work, and reduce the noise in the shell
* 10631: Get CircleCI to store Python coverage report
* 10631: Ensure coverage config file is copied into Docker container
..so that artefacts end up in the expected place for downstream processing
* 10631: Copy the coverage report to the CI host so it can be collected as an artefact
* 10631: Support CodeCov reporting
Allows for a query param to override the country (e.g. `?geo=de`).
Also removes the old Maxmind stuff since we're unlikely to use that
again, and add a GeoTemplateView class to make it easy to switch
template based on request country. Adds docs for all of this as well.
Have the app download the right data on startup. If that fails it won't
start the app and so the deployment will fail, but the existing
deployment will stay up.
The default list of latency buckets contains 17 entries. My proposal
is just 8.
5ca09db237/django_prometheus/middleware.py (L8-L26)
Also add an environment variable to deal with multiple wsgi processes
for prometheus, and use a version of django-prometheus that allows
us to set a metrics namespace.
The old way suddenly broke in that it no longer properly
included pip. This changes to the default recommended way
of creating a virtualenv in the Python 3.8 docs.
* Fix dict cache default timeout
* Add fallback strings and checking for translations in template
* Add ftl alias and ftl_lazy for use in py files
* Move git repos and add fluent to l10n_update
* Add percent_translated and required_messages to the l10n object
* Add management commands for converting en-US .lang file and templates
* Add command to convert en-US .lang file to .ftl file
* Using data from above command add a command that will convert a
template to use the .ftl strings
* Update the l10n_update command to update both external l10n repos
* Convert the /mission/ page to Fluent
* Add command to convert .lang translations to .ftl
* Copy en .ftl file when porting translations
* Removes the original string hash comments as well.
* Add Fluent docs.
* Add indentation and key sorting to metadata JSON output
* Add FTL file linting command
* Update mission template for FTL and remove FTL specific template
* Set l10n cache to 10s when DEBUG=True
* Fix and add tests for get_l10n_path
* Fix some style mistakes in docs and code
* Support multiline group comments for required strings
fix#7725, fix#7822, and fix#7726
The image we were using (`python:3-slim`) just switched to Python 3.8
and lxml was failing to build using that. This uses the much more
specific `python:3.7-slim-buster` tag so that this won't change for
us unexpectedly again.
Fix#7927
* Install Sphinx markdown deps
* Configure Sphinx with markdown support
* Switch to RTD Sphinx theme for local too
* Fix a bunch of Sphinx warnings for docs
* Separate docs deps into a requirements file for RTD to use
Steve Jalim
Rob Hudson
Alex Gibson
Jan Brasna
Paul McLanahan
Paul McLanahan
Alan
Ayushsunny
Axel Hecht
Chris Beaven