This makes it clearer what variables actually need to be set
locally, and which have been forgotten. It also makes the build
simpler, by removing the need to copy the .env-dist file.

This should be safe to apply, since .env-dist already got loaded by
default, just like .env now is. And it is still the case that
actual environment variables overwrite the ones in the .env file.

For non-Next.js setups (e.g. cron jobs or database migrations), I
switched to dotenv-flow. The regular dotenv explicitly avoids
inheritance [1], because it wants environment variables to be
specific to an environment. That was already not the case with most
of our environment variables, so the switch makes sense for us.

Next steps could be to remove unused variables from .env, and
possibly moving variables with local/stage-specific values to
.env.local.example, though that riskier, since environments might
depend on those being present.

[1]
https://www.npmjs.com/package/dotenv#should-i-have-multiple-env-files
This commit is contained in:
Vincent 2024-07-02 16:01:52 +02:00 коммит произвёл Vincent
Родитель b2775934a2
Коммит e7261bc8ca
23 изменённых файлов: 135 добавлений и 58 удалений

Просмотреть файл

@ -3,12 +3,6 @@ APP_ENV=local
SERVER_URL=http://localhost:6060
PORT=6060
NEXTAUTH_URL=http://localhost:6060
EMAIL_TEST_RECIPIENT=localmonitor20200827@mailinator.com
# see https://www.npmjs.com/package/mozlog
# default values are for dev/debug
MOZLOG_FMT=pretty
MOZLOG_LEVEL=debug
# 1: disables the dockerflow endpoints
# see: https://github.com/mozilla-services/Dockerflow#containerized-app-requirements
@ -38,8 +32,6 @@ AWS_REGION=
S3_BUCKET=
# Firefox Accounts OAuth
# leave FXA_ENABLED empty to disable FXA
FXA_ENABLED=
FXA_SETTINGS_URL=https://accounts.stage.mozaws.net/settings
OAUTH_CLIENT_ID=edd29a80019d61a1

80
.env.local.example Normal file
Просмотреть файл

@ -0,0 +1,80 @@
## Secrets (get these from a team member)
OAUTH_CLIENT_SECRET=
SMTP_URL=
# Have I Been Pwned setup
HIBP_KANON_API_TOKEN=
HIBP_API_TOKEN=
HIBP_NOTIFY_TOKEN=
# OneRep setup
ONEREP_API_KEY=
ONEREP_WEBHOOK_SECRET=
# Used for `npm run create-location-data`
AWS_ACCESS_KEY_ID=
AWS_SECRET_ACCESS_KEY=
## Settings you might want to customise
ADMINS=youraddress@mozilla.com,youraddress+anotheraccount@mozilla.com
DATABASE_URL="postgres://blurts:blurts@localhost:5432/blurts"
# Set these to be able to run end-to-end tests locally
E2E_TEST_ACCOUNT_EMAIL=
E2E_TEST_ACCOUNT_PASSWORD=
E2E_TEST_ACCOUNT_EMAIL_ZERO_BREACHES=
E2E_TEST_ACCOUNT_EMAIL_EXPOSURES_STARTED=
E2E_TEST_PAYPAL_LOGIN =
E2E_TEST_PAYPAL_PASSWORD =
## Good defaults for a local setup:
APP_ENV=local
SERVER_URL=http://localhost:6060
PORT=6060
NEXTAUTH_URL=http://localhost:6060
# A random string, see
# https://next-auth.js.org/configuration/options#secret
NEXTAUTH_SECRET=r7nKAKDWV0Bl53GHgQ/kA/EJCM2zvuH+8G3wZtwbXEA=
EMAIL_FROM="Firefox Monitor (local) breach-alerts@mozilla.com"
# Whether GA4 sends data or not. NOTE: must be set in build environment.
NEXT_PUBLIC_GA4_DEBUG_MODE=true
SUBSCRIPTION_BILLING_AMOUNT_YEARLY_US=13.37
SUBSCRIPTION_BILLING_AMOUNT_MONTHLY_US=42.42
# HIBP setup
HIBP_KANON_API_ROOT=https://enterprise.stage-api.haveibeenpwned.com
# Sentry setup
SENTRY_DSN=https://573f784b5cc7481ebf8c0c385d2ad776@o1069899.ingest.sentry.io/4504612374052864
NEXT_PUBLIC_SENTRY_DSN=https://573f784b5cc7481ebf8c0c385d2ad776@o1069899.ingest.sentry.io/4504612374052864
# SubPlat product and plan IDs, used for Plus subscriptions:
PREMIUM_PRODUCT_ID=prod_NErZh679W62lai
PREMIUM_PLAN_ID_MONTHLY_US=price_1MUNq0Kb9q6OnNsL4BoJgepf
PREMIUM_PLAN_ID_YEARLY_US=price_1MUNq0Kb9q6OnNsL4BoJgepf
# Mozilla Accounts URLs
FXA_SUBSCRIPTIONS_URL=https://accounts.stage.mozaws.net/subscriptions
FXA_SETTINGS_URL=https://accounts.stage.mozaws.net/settings
OAUTH_CLIENT_ID=edd29a80019d61a1
OAUTH_API_URI="https://api-accounts.stage.mozaws.net/v1"
# Set based on https://accounts.stage.mozaws.net/.well-known/openid-configuration
OAUTH_AUTHORIZATION_URI=https://accounts.stage.mozaws.net/authorization
OAUTH_PROFILE_URI=https://profile.stage.mozaws.net/v1/profile
OAUTH_TOKEN_URI=https://oauth.stage.mozaws.net/v1/token
OAUTH_ACCOUNT_URI = "https://oauth.accounts.firefox.com/v1"
# Which environment to run end-to-end tests against:
E2E_TEST_ENV=local
E2E_TEST_BASE_URL=http://localhost:6060
# Used for `npm run create-location-data`
AWS_REGION=us-east-1
S3_BUCKET=firefoxmonitor-dev-monitor-cdn-dev-static-website

1
.github/workflows/build.yaml поставляемый
Просмотреть файл

@ -15,7 +15,6 @@ jobs:
node-version: '22.3.x'
- run: npm ci
- run: npm run build-glean
- run: cp .env-dist .env
# Verify that the build (incl. type-checking) succeeds
# Upload sourcemaps to Sentry
- run: npm run build

3
.github/workflows/e2e_cron.yml поставляемый
Просмотреть файл

@ -25,8 +25,6 @@ jobs:
- name: Install dependencies
run: npm ci
- name: Copy env var
run: cp .env-dist .env
- name: Store Playwright's Version
run: |
# Get the current Playwright version listed in package.json
@ -59,7 +57,6 @@ jobs:
E2E_TEST_PAYPAL_LOGIN: ${{ secrets.E2E_TEST_PAYPAL_LOGIN }}
E2E_TEST_PAYPAL_PASSWORD: ${{ secrets.E2E_TEST_PAYPAL_PASSWORD }}
ADMINS: ${{ secrets.ADMINS }}
FXA_ENABLED: true
OAUTH_CLIENT_SECRET: ${{ secrets.OAUTH_CLIENT_SECRET }}
ONEREP_API_KEY: ${{ secrets.ONEREP_API_KEY }}
NEXTAUTH_SECRET: ${{ secrets.NEXTAUTH_SECRET }}

3
.github/workflows/e2e_pr.yml поставляемый
Просмотреть файл

@ -36,8 +36,6 @@ jobs:
- name: Install dependencies
run: npm ci
- name: Copy env var
run: cp .env-dist .env
- name: Setting up postgres
run: npm run db:migrate
env:
@ -74,7 +72,6 @@ jobs:
E2E_TEST_PAYPAL_LOGIN: ${{ secrets.E2E_TEST_PAYPAL_LOGIN }}
E2E_TEST_PAYPAL_PASSWORD: ${{ secrets.E2E_TEST_PAYPAL_PASSWORD }}
ADMINS: ${{ secrets.ADMINS }}
FXA_ENABLED: true
OAUTH_CLIENT_SECRET: ${{ secrets.OAUTH_CLIENT_SECRET }}
ONEREP_API_KEY: ${{ secrets.ONEREP_API_KEY }}
NEXTAUTH_SECRET: ${{ secrets.NEXTAUTH_SECRET }}

1
.github/workflows/lint.yaml поставляемый
Просмотреть файл

@ -16,7 +16,6 @@ jobs:
- run: npm ci
- run: npm run build-glean
- run: npm run build-nimbus
- run: cp .env-dist .env
# Mirror old linter from CircleCI, verifies that linter succeeds
- run: npm run lint
- run: node scripts/check-node-version-alignment.js

1
.github/workflows/unittests.yaml поставляемый
Просмотреть файл

@ -14,7 +14,6 @@ jobs:
with:
node-version: '22.3.x'
- run: npm ci
- run: cp .env-dist .env
- run: npm run build-glean
- run: npm run build-nimbus
- run: npm test

2
.gitignore поставляемый
Просмотреть файл

@ -1,7 +1,5 @@
node_modules
.node-version
.env
src/.env
coverage
.coveralls.yml
.DS_Store

Просмотреть файл

@ -19,7 +19,7 @@ COPY --chown=app:app . /app
RUN npm ci --audit=false && rm -rf ~app/.npm /tmp/*
COPY .env-dist ./.env
COPY .env ./.env
ARG NEXT_PUBLIC_GA4_DEBUG_MODE
ENV NEXT_PUBLIC_GA4_DEBUG_MODE=false

Просмотреть файл

@ -19,7 +19,7 @@ COPY --chown=app:app . /app
RUN npm ci --audit=false && rm -rf ~app/.npm /tmp/*
COPY .env-dist ./.env
COPY .env ./.env
ARG NEXT_PUBLIC_GA4_DEBUG_MODE
ENV NEXT_PUBLIC_GA4_DEBUG_MODE=false

Просмотреть файл

@ -53,8 +53,8 @@ To create the database tables ...
createdb test-blurts # for tests
```
2. Update the `DATABASE_URL` value in your `.env` file with your local db
credentials:
2. Update the `DATABASE_URL` value in your `.env.local` (see step 3 under
"Install") file with your local db credentials:
```
DATABASE_URL="postgres://<username>:<password>@localhost:<port>/blurts"
@ -81,10 +81,10 @@ To create the database tables ...
npm install
```
3. Copy the `.env-dist` file to `.env`:
3. Copy the `.env.local.example` file to `.env.local`:
```sh
cp .env-dist .env
cp .env.local.example .env.local
```
4. Install fluent linter (requires Python)
@ -115,7 +115,7 @@ To create the database tables ...
npm run create-location-data
```
8. Ensure that you have the right `env` variables/keys set in your `.env` file. You can retrieve the variables from the Firefox Monitor 1Password Vault, or through [Magic-Wormhole](https://magic-wormhole.readthedocs.io/en/latest/), by asking one of the our engineers.
8. Ensure that you have the right `env` variables/keys set in your `.env.local` file. You can retrieve the variables from the Firefox Monitor 1Password Vault, or through [Magic-Wormhole](https://magic-wormhole.readthedocs.io/en/latest/), by asking one of the our engineers.
### Run
@ -172,9 +172,6 @@ Monitor generates multiple emails that get sent to subscribers. To preview or te
### Mozilla accounts ("FxA", formerly known as Firefox accounts)
Subscribe with a Mozilla account is controlled via the `FXA_ENABLED`
environment variable. (See `.env-dist`)
The repo comes with a development FxA oauth app pre-configured in `.env`, which
should work fine running the app on http://localhost:6060. You'll need to get
the `OAUTH_CLIENT_SECRET` value from a team member or someone in #fxmonitor-engineering.
@ -222,9 +219,3 @@ We use GCP Cloudrun for dev review – official stage and production apps are bu
_**TODO:** add full deploy process similar to Relay_
_**TODO:** consider whether we can re-enable Heroku Review Apps_
## Preserve sessions in local development
Sessions by default are stored in-memory, which means that when the server restarts (e.g. because you made a code change), you will have to log in again.
To avoid this hassle, you can install and run [Redis](https://redis.io/), which by default runs on `redis://localhost:6379`. Use that value for `REDIS_URL` in your `.env` file to preserve your sessions across server restarts.

Просмотреть файл

@ -8,7 +8,7 @@ Telemetry code is generated from `metrics.yml` by running `npm run build-glean`,
## Debugging
Make sure you are testing in a browser that does not set DoNotTrack, and that the value of `APP_ENV` in your `.env` file is `local`.
Make sure you are testing in a browser that does not set DoNotTrack, and that the value of `APP_ENV` in your `.env.local` file is `local`.
### Glean

Просмотреть файл

@ -72,7 +72,7 @@ added.
We're only using a test utility from this package (see the comments in
`/jest.setup.ts`), so if the tests pass, this upgrade was probably fine.
### `@playwright/test` and `dotenv`
### `@playwright/test` and `dotenv-flow`
Our Playwright tests run as two GitHub Actions Workflows: [PR e2e tests][] and
[cron e2e tests][]: the former runs a small subset of the end-to-end tests for PRs, the latter runs a more extensive suite once a day from the `main` branch.

Просмотреть файл

@ -36,3 +36,14 @@ global.IntersectionObserver = jest.fn();
defaultFallbackInView(false);
global.TextEncoder = TextEncoder;
// Jest doesn't like the top-level await in AppConstants, so we mock it. In
// time we can hopefully phase out the entire file and just use dotenv-flow
// and process.env directly.
jest.mock("./src/appConstants.js", () => {
// eslint-disable-next-line @typescript-eslint/no-var-requires
require("dotenv-flow").config();
return {
...process.env,
};
});

Просмотреть файл

@ -16,6 +16,6 @@
publish = "storybook-static/"
# Default build command.
command = "npm ci; cp .env-dist .env; npm run build-storybook"
command = "npm ci; npm run build-storybook"
environment = { NODE_VERSION = "22.3.0", NPM_VERSION = "10.8.0" }

14
package-lock.json сгенерированный
Просмотреть файл

@ -29,7 +29,7 @@
"@types/react": "^18.2.79",
"@types/react-dom": "^18.2.19",
"canvas-confetti": "^1.9.3",
"dotenv": "^16.4.5",
"dotenv-flow": "^4.1.0",
"eslint-config-next": "^14.2.4",
"jsdom": "^24.1.0",
"jsonwebtoken": "^9.0.2",
@ -16306,6 +16306,18 @@
"node": ">=12"
}
},
"node_modules/dotenv-flow": {
"version": "4.1.0",
"resolved": "https://registry.npmjs.org/dotenv-flow/-/dotenv-flow-4.1.0.tgz",
"integrity": "sha512-0cwP9jpQBQfyHwvE0cRhraZMkdV45TQedA8AAUZMsFzvmLcQyc1HPv+oX0OOYwLFjIlvgVepQ+WuQHbqDaHJZg==",
"license": "MIT",
"dependencies": {
"dotenv": "^16.0.0"
},
"engines": {
"node": ">= 12.0.0"
}
},
"node_modules/duplexify": {
"version": "4.1.2",
"resolved": "https://registry.npmjs.org/duplexify/-/duplexify-4.1.2.tgz",

Просмотреть файл

@ -27,8 +27,8 @@
"cron:db-pull-breaches": "node src/scripts/syncBreaches.js",
"cron:remote-settings-pull-breaches": "node scripts/updatebreaches.js",
"cron:onerep-limits-alert": "node src/scripts/onerepStatsAlert.js",
"db:migrate": "node -r dotenv/config node_modules/knex/bin/cli.js migrate:latest --knexfile src/db/knexfile.js",
"db:rollback": "node -r dotenv/config node_modules/knex/bin/cli.js migrate:rollback --knexfile src/db/knexfile.js",
"db:migrate": "node -r dotenv-flow/config node_modules/knex/bin/cli.js migrate:latest --knexfile src/db/knexfile.js",
"db:rollback": "node -r dotenv-flow/config node_modules/knex/bin/cli.js migrate:rollback --knexfile src/db/knexfile.js",
"prepare": "husky",
"storybook": "storybook dev -p 6006",
"build-storybook": "npm run build-glean && npm run build-nimbus && storybook build",
@ -75,7 +75,7 @@
"@types/react": "^18.2.79",
"@types/react-dom": "^18.2.19",
"canvas-confetti": "^1.9.3",
"dotenv": "^16.4.5",
"dotenv-flow": "^4.1.0",
"eslint-config-next": "^14.2.4",
"jsdom": "^24.1.0",
"jsonwebtoken": "^9.0.2",

Просмотреть файл

@ -5,10 +5,10 @@
import { defineConfig, devices } from '@playwright/test'
/**
* Read environment variables from file.
* https://github.com/motdotla/dotenv
* https://www.npmjs.com/package/dotenv-flow
*/
import * as dotenv from 'dotenv'
dotenv.config()
import * as dotenvFlow from 'dotenv-flow'
dotenvFlow.config()
/**
* @see https://playwright.dev/docs/test-configuration
*/

Просмотреть файл

@ -59,7 +59,7 @@ export function createGetL10nBundles(
const languages = acceptedLanguages(acceptLangHeader);
const supportedLocales = process.env.SUPPORTED_LOCALES?.split(",");
const filteredLocales =
// `SUPPORTED_LOCALES` is set in the `.env-dist` file, so it'll always
// `SUPPORTED_LOCALES` is set in the `.env` file, so it'll always
// be available when running tests.
/* c8 ignore next 2 */
typeof supportedLocales === "undefined"

Просмотреть файл

@ -3,8 +3,14 @@
* file, You can obtain one at http://mozilla.org/MPL/2.0/. */
// TODO: these vars were copy/pasted from the old app-constants.js and should be cleaned up
import * as dotenv from 'dotenv'
dotenv.config()
if (typeof process.env.NEXT_RUNTIME === "undefined") {
// Next.js already loads env vars by itself, and dotenv-flow will throw an
// error if loaded in that context (about `fs` not existing), so only load
// it if we're not running in a Next.js-context (e.g. cron jobs):
const dotenvFlow = await import("dotenv-flow");
dotenvFlow.config();
}
const requiredEnvVars = [
'ADMINS',
@ -12,15 +18,12 @@ const requiredEnvVars = [
'DATABASE_URL',
'DELETE_UNVERIFIED_SUBSCRIBERS_TIMER',
'EMAIL_FROM',
'FXA_ENABLED',
'HIBP_API_ROOT',
'HIBP_KANON_API_ROOT',
'HIBP_KANON_API_TOKEN',
'HIBP_NOTIFY_TOKEN',
'HIBP_THROTTLE_DELAY',
'HIBP_THROTTLE_MAX_TRIES',
'MOZLOG_FMT',
'MOZLOG_LEVEL',
'FXA_SETTINGS_URL',
'NODE_ENV',
'OAUTH_ACCOUNT_URI',
@ -36,7 +39,6 @@ const requiredEnvVars = [
]
const optionalEnvVars = [
'EMAIL_TEST_RECIPIENT',
'FX_REMOTE_SETTINGS_WRITER_PASS',
'FX_REMOTE_SETTINGS_WRITER_SERVER',
'FX_REMOTE_SETTINGS_WRITER_USER',

Просмотреть файл

@ -14,13 +14,13 @@ npm install
npx playwright install
```
#### Generate .env from .env-dist
#### Generate .env.local from .env.local.example
```sh
cp .env-dist .env
cp .env.local.example .env.local
```
#### Add some additional values to the .env file
#### Add some additional values to the .env.local file
`E2E_TEST_ENV=<'local' | 'dev' | 'stage'>` stage is default, for prod please use GA
@ -36,7 +36,7 @@ Reach out to @mansaj for the paypal login. The e2e test email and password shoul
#### Verify helpers.js to make sure correct urls are being used
#### e2e runs on localhost:6060 by default, in order to run against other envs, update the `E2E_TEST_ENV` variable in the .env file
#### e2e runs on localhost:6060 by default, in order to run against other envs, update the `E2E_TEST_ENV` variable in the .env.local file
#### To debug a test run

Просмотреть файл

@ -5,7 +5,7 @@
import { createWriteStream, existsSync } from "fs";
import { Readable } from "stream";
import { finished } from "stream/promises";
import "dotenv/config";
import "dotenv-flow/config";
const dataPath = "./locationAutocompleteData.json";

Просмотреть файл

@ -4,7 +4,7 @@
import { Upload } from "@aws-sdk/lib-storage";
import { S3 } from "@aws-sdk/client-s3";
import "dotenv/config";
import "dotenv-flow/config";
const accessKeyId = process.env.AWS_ACCESS_KEY_ID;
const secretAccessKey = process.env.AWS_SECRET_ACCESS_KEY;