Simplify docker invoke tasks (#3673)

* Add a config file for invoke. Echo commands before running them * Simplify docker related task + remove the double .env file * update docs * Forgot to change one line of the docs * Update tasks.py typo Co-Authored-By: Pomax <pomax@nihongoresources.com>
2019-09-23 10:59:38 +02:00 · 2019-09-23 10:59:38 +02:00 · 0a595d9698
--- a/README.md
+++ b/README.md
@ -66,9 +66,9 @@ In addition to the code tests there are also visual regression tests, located in
 - Install [Docker Desktop](https://www.docker.com/products/docker-desktop) (macOS and Windows). For Linux users: install [Docker CE](https://docs.docker.com/install/#supported-platforms) and [Docker Compose](https://docs.docker.com/compose/install/). If you don't want to create a Docker account, direct links to download can be found [in this issue](https://github.com/docker/docker.github.io/issues/6910),
 - [Check your install](https://docs.docker.com/get-started/#test-docker-version) by running `docker run hello-world`,
 - If relevant: delete your node_modules directory (`rm -rf node_modules`). It's not necessary, but it speeds up the install.
- Run `invoke docker-setup` ([install invoke](http://www.pyinvoke.org/installing.html) if you don't have it yet). If you're running on Windows, you need to run `docker-compose --rm pipenv run python network-api/manage.py createsuperuser` when the setup is finished.
+- Run `invoke docker-new-env` ([install invoke](http://www.pyinvoke.org/installing.html) if you don't have it yet). If you're running on Windows, you need to run `docker-compose --rm pipenv run python network-api/manage.py createsuperuser` when the setup is finished.

-This task is copying your `.env` to the new `.docker.env` that is in charge of managing your environment variables while running Docker. The installation will take a few minutes: you need to download images from the Docker Hub, install JS and Python dependencies, create fake data, migrate your database, etc.
+This task is creating a `.env` that is in charge of managing your environment variables while running Docker. The installation will take a few minutes: you need to download images from the Docker Hub, install JS and Python dependencies, create fake data, migrate your database, etc.

 When it's done, run `docker-compose up`, wait until the static files to be built, and go to `0.0.0.0:8000`. You should have a local working version of the foundation site with fake data. When you want to stop, do `^C` to shut down your containers.

--- a/docker-compose.yml
+++ b/docker-compose.yml
@ -28,10 +28,7 @@ services:
      context: .
      dockerfile: ./dockerfiles/Dockerfile.python
    env_file:
-      - ".docker.env"
-    environment:
-      # Prevents pipenv from loading env files (python dotenv and docker-compose compatibility issue).
-      - PIPENV_DONT_LOAD_ENV=1
+      - ".env"
    command: pipenv run python network-api/manage.py runserver 0.0.0.0:8000
    ports:
      - "8000:8000"
--- a/docs/local_development_with_docker.md
+++ b/docs/local_development_with_docker.md
@ -11,7 +11,7 @@ This documentation is composed of three main sections:
 To interact with the project, you can use [docker](https://docs.docker.com/engine/reference/commandline/cli/) and [docker-compose](https://docs.docker.com/compose/reference/overview/) CLIs or use shortcuts with invoke.

 The general workflow is:
- Install the project with `invoke docker-setup`,
+- Install the project with `invoke docker-new-env`,
 - Run the project with `docker-compose up`,
 - Use invoke commands for frequent development tasks (database migrations, dependencies install, run tests, etc),
 - After doing a `git pull`, keep your clone up to date by running `invoke docker-catchup`.
@ -27,11 +27,10 @@ The general workflow is:
  docker-makemigrations              Creates new migration(s) for apps
  docker-manage                      Shorthand to manage.py. inv docker.manage "[COMMAND] [ARG]"
  docker-migrate                     Updates database schema
-  docker-npm                         Shorthand to npm. inv docker.npm "[COMMAND] [ARG]"
  docker-new-db                      Delete your database and create a new one with fake data
+  docker-new-env                     Get a new dev environment and a new database with fake data
+  docker-npm                         Shorthand to npm. inv docker.npm "[COMMAND] [ARG]"
  docker-pipenv                      Shorthand to pipenv. inv docker.pipenv "[COMMAND] [ARG]"
-  docker-setup                       Prepare your dev environment after a fresh git clone
-  docker_switching_branch            Get a new database with fake data and rebuild images
  docker-test-node                   Run node tests
  docker-test-python                 Run python tests
 ```
@ -173,6 +172,5 @@ We still use all those tools with Docker. The major difference is that `npm` and
 ### Can I use Docker in parallel with the old way of running the foundation site?

 Short answer is yes but:
- you will have two different databases
- you will have two files to manage your environments variables (`.env` and `.env.docker`),
+- you will have two different databases.
 - those two environment won't share their dependencies: you will have to maintain and update both of them.
--- a/env.default
+++ b/env.default
@ -14,15 +14,15 @@ ALLOWED_HOSTS=*
 ASSET_DOMAIN=network.mofoprod.net
 CONTENT_TYPE_NO_SNIFF=True
 CORS_WHITELIST=*
-CORS_REGEX_WHITELIST=""
-DATABASE_URL=""
+CORS_REGEX_WHITELIST=
+DATABASE_URL=
 DEBUG=True
 DISABLE_DEBUG_TOOLBAR=True
 DJANGO_SECRET_KEY=secret
 LOAD_FIXTURE=False
 SET_HSTS=False
-SOCIAL_AUTH_GOOGLE_OAUTH2_KEY=""
-SOCIAL_AUTH_GOOGLE_OAUTH2_SECRET=""
+SOCIAL_AUTH_GOOGLE_OAUTH2_KEY=
+SOCIAL_AUTH_GOOGLE_OAUTH2_SECRET=
 SOCIAL_AUTH_LOGIN_REDIRECT_URL=http://localhost:5000/soc/complete/google-oauth2/
 SSL_REDIRECT=False
 USE_S3=False
@ -32,46 +32,46 @@ XSS_PROTECTION=True

 # SQS config

-AWS_SQS_ACCESS_KEY_ID=""
-AWS_SQS_SECRET_ACCESS_KEY=""
-AWS_SQS_REGION=""
-PETITION_SQS_QUEUE_URL=""
+AWS_SQS_ACCESS_KEY_ID=
+AWS_SQS_SECRET_ACCESS_KEY=
+AWS_SQS_REGION=
+PETITION_SQS_QUEUE_URL=

-CRM_AWS_SQS_ACCESS_KEY_ID=""
-CRM_AWS_SQS_SECRET_ACCESS_KEY=""
-CRM_AWS_SQS_REGION=""
-CRM_PETITION_SQS_QUEUE_URL=""
+CRM_AWS_SQS_ACCESS_KEY_ID=
+CRM_AWS_SQS_SECRET_ACCESS_KEY=
+CRM_AWS_SQS_REGION=
+CRM_PETITION_SQS_QUEUE_URL=


 # CSP config

-CSP_DEFAULT_SRC="'none'"
+CSP_DEFAULT_SRC='none'
 CSP_SCRIPT_SRC='self' 'unsafe-inline' cdn.optimizely.com https://www.google-analytics.com/analytics.js https://*.shpg.org/ http://ajax.googleapis.com/ajax/libs/jquery/2.2.4/jquery.min.js https://airtable.com https://platform.twitter.com https://cdn.syndication.twimg.com
 CSP_STYLE_SRC='self' 'unsafe-inline' code.cdn.mozilla.net fonts.googleapis.com https://platform.twitter.com
 CSP_IMG_SRC=* data:
 CSP_FONT_SRC='self' fonts.gstatic.com fonts.googleapis.com code.cdn.mozilla.net
 CSP_CONNECT_SRC=*
-CSP_MEDIA_SRC="'self'"
+CSP_MEDIA_SRC='self'
 CSP_CHILD_SRC='self' https://www.youtube.com https://s3.amazonaws.com
 CSP_FORM_ACTION='self' https://www.mozilla.org/en-US/newsletter/
 CSP_FRAME_SRC=https://airtable.com https://platform.twitter.com

 # TEST ENVIRONMENT VALUES

-PETITION_TEST_CAMPAIGN_ID=""
+PETITION_TEST_CAMPAIGN_ID=

 # REVIEW APPS SLACK BOT

-GITHUB_TOKEN=""
-SLACK_WEBHOOK_RA=""
+GITHUB_TOKEN=
+SLACK_WEBHOOK_RA=

 # BUYER'S GUIDE Configuration
-CORAL_TALK_SERVER_URL=""
-PNI_STATS_DB_URL=""
-CORAL_TALK_API_TOKEN=""
+CORAL_TALK_SERVER_URL=
+PNI_STATS_DB_URL=
+CORAL_TALK_API_TOKEN=

 # Cloudinary configuration
 USE_CLOUDINARY=False
-CLOUDINARY_CLOUD_NAME=""
-CLOUDINARY_API_KEY=""
-CLOUDINARY_API_SECRET=""
+CLOUDINARY_CLOUD_NAME=
+CLOUDINARY_API_KEY=
+CLOUDINARY_API_SECRET=
--- a/invoke.yaml
+++ b/invoke.yaml
@ -0,0 +1,2 @@
+run:
+    echo: true
--- a/tasks.py
+++ b/tasks.py
@ -19,10 +19,10 @@ else:


 def create_docker_env_file(env_file):
-    """Create a docker .env from an existing env file"""
+    """Create or update an .env to work with a docker environment"""
    with open(env_file, 'r') as f:
        env_vars = f.read()
-    # remove the double quote for docker-compose and python_dotenv compatibility reasons
+    # We need to strip the quotes because Docker-compose considers them as part of the env value.
    env_vars = env_vars.replace('"', '')
    # update the DATABASE_URL env
    new_db_url = "DATABASE_URL=postgres://postgres@postgres:5432/postgres"
@ -34,7 +34,7 @@ def create_docker_env_file(env_file):
    env_vars = env_vars.replace(old_hosts.group(0), new_hosts)

    # create the new env file
-    with open('.docker.env', 'w') as f:
+    with open('.env', 'w') as f:
        f.write(env_vars)


@ -82,9 +82,9 @@ def l10n_update(ctx):
@task
 def test(ctx):
    """Run tests"""
-    print("Running flake8")
+    print("* Running flake8")
    ctx.run(f"pipenv run flake8 tasks.py network-api", **PLATFORM_ARG)
-    print("Running tests")
+    print("* Running tests")
    manage(ctx, "test")


@ -92,29 +92,29 @@ def test(ctx):
 def setup(ctx):
    """Prepare your dev environment after a fresh git clone"""
    with ctx.cd(ROOT):
-        print("Setting default environment variables.")
+        print("* Setting default environment variables.")
        if os.path.isfile(".env"):
-            print("Keeping your existing .env")
+            print("* Keeping your existing .env")
        else:
-            print("Creating a new .env")
+            print("* Creating a new .env")
            copy("env.default", ".env")
-        print("Installing npm dependencies and build.")
+        print("* Installing npm dependencies and build.")
        ctx.run("npm install && npm run build")
-        print("Installing Python dependencies.")
+        print("* Installing Python dependencies.")
        ctx.run("pipenv install --dev")
-        print("Applying database migrations.")
+        print("* Applying database migrations.")
        migrate(ctx)
-        print("Updating localizable fields")
+        print("* Updating localizable fields")
        l10n_sync(ctx)
        l10n_update(ctx)
-        print("Creating fake data")
+        print("* Creating fake data")
        manage(ctx, "load_fake_data")
-        print("Updating block information")
+        print("* Updating block information")
        manage(ctx, "block_inventory")

        # Windows doesn't support pty, skipping this step
        if platform == 'win32':
-            print("All done!\n"
+            print("\nAll done!\n"
                  "To create an admin user: pipenv run python network-api/manage.py createsuperuser\n"
                  "To start your dev server: inv runserver")
        else:
@ -126,21 +126,42 @@ def setup(ctx):
@task(aliases=["catchup"])
 def catch_up(ctx):
    """Install dependencies and apply migrations"""
-    print("Installing npm dependencies and build.")
+    print("* Installing npm dependencies and build.")
    ctx.run("npm install && npm run build")
-    print("Installing Python dependencies.")
+    print("* Installing Python dependencies.")
    ctx.run("pipenv install --dev")
-    print("Applying database migrations.")
+    print("* Applying database migrations.")
    migrate(ctx)
-    print("Updating localizable fields")
+    print("* Updating localizable fields")
    l10n_sync(ctx)
    l10n_update(ctx)
-    print("Updating block information")
+    print("* Updating block information")
    manage(ctx, "block_inventory")


 # Tasks with Docker

+def docker_l10n_block_inventory(ctx):
+    print("* Updating localizable fields")
+    docker_l10n_sync(ctx)
+    docker_l10n_update(ctx)
+    print("* Updating block information")
+    docker_manage(ctx, "block_inventory")
+
+
+def docker_create_super_user(ctx):
+    # Windows doesn't support pty, skipping this step
+    if platform == 'win32':
+        print("\nPTY is not supported on Windows.\n"
+              "To create an admin user:\n"
+              "docker-compose run --rm backend pipenv run python network-api/manage.py createsuperuser\n")
+    else:
+        print("* Creating superuser.")
+        ctx.run(
+            "docker-compose run --rm backend pipenv run python network-api/manage.py createsuperuser",
+            pty=True
+        )
+

@task
 def docker_manage(ctx, command):
@ -190,103 +211,66 @@ def docker_l10n_update(ctx):
@task
 def docker_test_python(ctx):
    """Run python tests"""
-    print("Running flake8")
+    print("* Running flake8")
    ctx.run("docker-compose run --rm backend pipenv run flake8 tasks.py network-api", **PLATFORM_ARG)
-    print("Running tests")
+    print("* Running tests")
    docker_manage(ctx, "test")


@task
 def docker_test_node(ctx):
    """Run node tests"""
-    print("Running tests")
+    print("* Running tests")
    ctx.run("docker-compose run --rm watch-static-files npm run test", **PLATFORM_ARG)


-@task
-def docker_switching_branch(ctx):
-    """Get a new database with fake data and rebuild images"""
-    print("Stopping services first")
-    ctx.run("docker-compose down")
-    print("Deleting database")
-    ctx.run("docker volume rm foundationmozillaorg_postgres_data")
-    print("Rebuilding images and install dependencies")
-    ctx.run("docker-compose build")
-    print("Applying database migrations.")
-    docker_migrate(ctx)
-    print("Creating fake data")
-    docker_manage(ctx, "load_fake_data")
-    print("Updating localizable fields")
-    docker_l10n_sync(ctx)
-    docker_l10n_update(ctx)
-    print("Updating block information")
-    docker_manage(ctx, "block_inventory")
-
-
@task
 def docker_new_db(ctx):
    """Delete your database and create a new one with fake data"""
-    print("Stopping services first")
+    print("* Stopping services first")
    ctx.run("docker-compose down")
-    print("Deleting database")
+    print("* Deleting database")
    ctx.run("docker volume rm foundationmozillaorg_postgres_data")
-    print("Applying database migrations.")
+    print("* Applying database migrations.")
    docker_migrate(ctx)
-    print("Creating fake data")
+    print("* Creating fake data")
    docker_manage(ctx, "load_fake_data")
-    print("Updating localizable fields")
-    docker_l10n_sync(ctx)
-    docker_l10n_update(ctx)
-    print("Updating block information")
-    docker_manage(ctx, "block_inventory")
+    docker_l10n_block_inventory(ctx)
+    docker_create_super_user(ctx)


@task(aliases=["docker-catchup"])
 def docker_catch_up(ctx):
    """Rebuild images and apply migrations"""
-    print("Stopping services first")
+    print("* Stopping services first")
    ctx.run("docker-compose down")
-    print("Rebuilding images and install dependencies")
+    print("* Rebuilding images and install dependencies")
    ctx.run("docker-compose build")
-    print("Applying database migrations.")
+    print("* Applying database migrations.")
    docker_migrate(ctx)
-    print("Updating localizable fields")
-    docker_l10n_sync(ctx)
-    docker_l10n_update(ctx)
-    print("Updating block information")
-    docker_manage(ctx, "block_inventory")
+    docker_l10n_block_inventory(ctx)


@task
-def docker_setup(ctx):
-    """Prepare your dev environment after a fresh git clone"""
+def docker_new_env(ctx):
+    """Get a new dev environment and a new database with fake data"""
    with ctx.cd(ROOT):
-        print("Setting default environment variables.")
+        print("* Setting default environment variables")
        if os.path.isfile(".env"):
-            print("Copying your existing .env to .docker.env")
+            print("* Stripping quotes and making sure your DATABASE_URL and ALLOWED_HOSTS are properly setup")
            create_docker_env_file(".env")
        else:
-            if os.path.isfile('.docker.env'):
-                print("Keeping your existing .docker.env.")
-            else:
-                print("Creating a new .docker.env")
-                create_docker_env_file("env.default")
-        print("Building Docker images")
-        ctx.run("docker-compose build", **PLATFORM_ARG)
-        print("Setting up a clean database.")
-        docker_new_db(ctx)
+            print("* Creating a new .env")
+            create_docker_env_file("env.default")
+        print("* Stopping project's containers and delete volumes if necessary")
+        ctx.run("docker-compose down --volumes")
+        print("* Building Docker images")
+        ctx.run("docker-compose build --no-cache", **PLATFORM_ARG)
+        print("* Applying database migrations.")
+        docker_migrate(ctx)
+        print("* Creating fake data")
+        docker_manage(ctx, "load_fake_data")
+        docker_l10n_block_inventory(ctx)
+        docker_create_super_user(ctx)

-        # Windows doesn't support pty, skipping this step
-        if platform == 'win32':
-            print("All done!\n"
-                  "To create an admin user:\n"
-                  "docker-compose run --rm backend pipenv run python network-api/manage.py createsuperuser\n"
-                  "To start your dev server:\n"
-                  "docker-compose up")
-        else:
-            print("Creating superuser.")
-            ctx.run(
-                "docker-compose run --rm backend pipenv run python network-api/manage.py createsuperuser",
-                pty=True
-            )
-            print("All done! To start your dev server, run the following:\n docker-compose up")
+        print("\n* Start your dev server with:\n docker-compose up")