* More efficient utilization of github action cache
* Set cache from/to only in CI
* Update Dockerfile
Co-authored-by: Mathieu Pillard <diox@users.noreply.github.com>
---------
Co-authored-by: Mathieu Pillard <diox@users.noreply.github.com>
* Fix Docker build:
- get correct Image ID from the build.
- set docker tag correctly supporting tag and digest
- add concurrency limit to github action ci jobs
* TMP: remove easy way to get build digest as it is not really intended for local dev
* Update .circleci/config.yml
Co-authored-by: Mathieu Pillard <diox@users.noreply.github.com>
* TMP: recomment build
---------
Co-authored-by: Mathieu Pillard <diox@users.noreply.github.com>
* TMP: refactor buildx bake definition and test
* TMP: better initialization and organization of make file env vars (docker relevant +)
* TMP: cleaner code.
* TMP: remove build
* Infer superuser credentials from git config
* Load service data via portable and persistent data volumes
Load storage via persistent data volume (host bound)
Make up/down commands to easily recreate local environments
* TMP: add redis volume too
* TMP: better control over volume state
* TMP: remove dead code
* TMP: more dead code gone
* TMP: add wait for mysql before restoring snapshots
* Use consistent and robust mapping of UID between host and container:
- collapse UID/GID to one value on host and container
- rename host value to HOST_UID for clarity and debugability
- rename fix_olympia_user to entrypoint and set as docker compose entrypoint
- uwsgi config to use runtime value of olympia UID/GID
* Docker compose fail after 5 attempts to start
* TMP: update outdated comment
* TMP: comment the entrypoint
* Extract logic to load verison.json to utils
* Reimplement version.json check as django check
* Better defaults for version commit/build
* Extract django check to make command
* Generate version.json as part of docker build:
- remove git dependency from version.json generation
- use only version.json file for sentry release with soft/hard fail
* TMP: remove invalid comment
* TMP: Prefix branch tags
* TMP: re-comment the ci build in circle
* Reusable Docker run action (Bunch of other stuff here to make it work)
* extract locales
* remove extraneous log
---------
Co-authored-by: Mozilla Add-ons Robot <addons-dev-automation+github@mozilla.com>
* local docker build command
* Move copy . to after installing packages (prevents always re-installing)
* Enable buildkit cache mount to cache pip/npm dependencies
- cache npm/pip dependencies across builds
- add clear logging to npm install for cache hit/miss
- move copy npm files to update_assets
* Split build into stages
- splitting to stages offers better caching of layers and more efficient use of disk/time
- The initial gains will be with better caching of locale compilation, but will expand as we can move more logic from the final stage
TODO: split up apt depedencies to specific stages, move update_assets to pre-final stage to prevent re-running on every build with a * file change.
* Document the docker file and build process
* update docs
This updates various things…
* make use o `--user` argument of pip and `$PYTHONUSERBASE` to install python libraries in the correct places where the `olympia` user has permissions
* Add a new `rootshell` command
Fixes#12366
* Correct if statement
* Use separate tag for easier testing
* Try running directly with user olympia
* Small cleanup
* Add docs
* Fix comment
* Add 'rootshell' command
* Build circleci for local dev
* Fix pip execution by using 'python -m' to fetch the correct pip
* Pick up upstream user/group for builds and implement PYTHONUSERBASE and PIP_USER
* Correctly pass USER_ID and GROUP_ID to Dockerfile when building
* Small cleanups
* Build our test branch
* Fall back to manual uploads for testing
* Cleanup
* Revert back to :latest tag
* move all the make commands that operate on a single container to Makefile-docker
* add default user to Dockerfile
* revert the PYTHON_COMMAND var change
* Fix uwsgi autoreload.
This will scan python modules for modifications every 3 seconds. It most
certainly will still be more efficient than Django's autoreload.
Fixes#11990
* re-add vaccum
* Use watchmedo to reload properly
* Fix docs for python 3
* Make travis behave like it's inside our container, fix tox/makefile accordingly
* Fix docs build by choosing another label than _search (which is already used)
* Fix codestyle linter issues
* Remove extra force_bytes import
* Add ignore for int|null in docs as it's used in a couple places and it's valid
* Fix more docs build failures in reviewers.rst
Also build docs with -W by default to make future docs breakage
easier to spot locally.
* Leave -f Makefile-docker for UI Tests as they are not running from the standard container
* Make also use of setting PYTHONDONTWRITEBYTECODE inside the docker
image to reduce it's size considerably
* Remove some outdated code-parts
* Rename `olympia` inside our worker image to `olympia-worker`
* remove all build and cache folders after every `make update_deps`
(which might help avoid any more package update/install ambiquities)
Fixes#7518
* Also let the dot in py.test die so that upstream is happy again :)
* Update tox to 2.9.1
* Make sure that our urlconf is reset *after* we set MEDIA_ROOT
* Adapt file extraction handling to new TMP_PATH prefixes
* Less dump_apps hacks since the apps.json file isn't present at the
beginning of any test now thanks to temporary paths for every separate
test
* Rewrite read-only-mode tests to cope with pytest settings fixture
* Remove atexit hack from settings_test
This will probably make things a lot easier for ui-tests as well.
Fixes#7205Fixes#7206
* Update celery, config and dependencies to 4.x
* Remove usage of TaskSet, replace with group(), update `ampq`
* Enable redis on travis for better celery 4.x testing
* Fix lib/celery_tests
* Correct test testing email sending retrying on errors
* Fix celery subtask mocking errors.
* Fix celery broker url env variable
* Integrate watchdog for celery autoreload
* Ensure dependencies are updated for worker and web container.
* Restart all services after the dependency update
* Remove explicit celery loader environment variable.
* Restart all services in web and worker after running 'update_docker'
* Increase sleep time a bit
Kevin Meinhardt
Mathieu Pillard
Andreas Wagner
Christopher Grebs
William Durand
Andrew Williamson
Benjamin Forehand Jr
Mathieu Pillard