Because:
- We wanted to run a few preliminary checks before proceeding to more
expensive CI jobs. Checks include:
- Compiling typescript in commonly referenced workspace packages
- Linting code that has changed
- Executing Unit Tests for code that has changed
- We wanted to partition test operations into unit tests, and
integration tests. Unit tests can be run relatively quickly and
require no additional infrastructure. Integration tests require
additional infrastructure and generally have longer execution
times. Now that jobs are blocked from running until preliminary
checks pass, one of which is unit tests, it is important to draw a
distinction between these two types of tests.
- We want to avoid unnecessary yarn installs and typescript
compilations, which are time consuming.
- We want to make sure that test results are published and failing tests
can be easily viewed in the CI.
This Commit:
- Creates a build-and-validate job in the CI that builds, lints, and
unit tests code prior to running any other jobs.
- Creates unit-test job in CI config
- Creates integration-test job in CI config
- Removes redundant calls to compile workspace packages. These
are now built up front, cached, and restored as needed for future
runs.
- Extends the create-lists script functionality to generate commands
that can be executed with the parallel command.
- Removes unnecessary yarn install operations. Invoking yarn workspace
focus results in a yarn install. In the case of running tests this is largely
unnecessary, because we already do a yarn install in the base-install
step.
- Make sure test results are exported as junit xml so the CI can report
back on tests that were failing. This was done for a couple workspace
packages, but many were lacking the capability. All test:unit and
test:integration npm scripts now export this data.
- Fixes the following issues encountered along the way:
- Adds logs to monitor heap usage of jest tests. Some
jest tests are still using a lot of memory.
- Moves a few slow / long running tests from unit test to
integration tests.
- Ensures that jest.transform for ts-jest is always instructed
to have the config option isolateModules is set to true. This
definitely decreases memory overhead and resolves some
of the OOM errors we were hitting. It was configured in
some places but not everywhere.
- Exports test results files for all tests
- Exports all test artifacts
- Uses gnu parallel to run tests in parallel. Turns out yarn
workspaces foreach would give a false positive when an OOM
was encountered. Fortunately, the parallel command offered an
acceptable work around, and even offers some nice features
like the load argument, which allows to control test execution a
bit more efficiently.
Because:
- We want to separate unit tests and integrations tests
This Commit:
- Tags all unit tests, i.e. tests that can run without infrastructure and in a fairly short amount of time, with #unit.
- Adds a test:unit npm command to each package
- Adds a test:integration npm command to each package, and adds '#integration' tags to jest tests since jest has no --invert option like mocha.
* feat(cli): add 'yarn delete services' command
sometimes you may want to leave all your DBs running but completely start over with fresh services. The pm2 restart command doesn't fully reset the environment and sometimes that's what you want. Delete lets us remove the service from pm2 so we can start it fresh.
* chore(ci): slow down pm2 startup
I've noticed ci failures where the services don't even get started up completely happening more frequently. During the brief time that circleci showed resource graphs the pm2 startup pegged both cpu and memory. If we wait for the services that have a lot of building at startup to finish before starting the next service maybe we'll have better stability.
* chore(tests): remove support-panel from functional test startup
chore(deps): updated aws-sdk
chore(deps): remove raven and update @sentry/*
chore(deps): updated mysql deps
chore(deps): updated eslint-plugin-fxa
chore(deps): updated bluebird
chore(deps): updated request
chore(deps): use jest 24.9.0 (same as react-scripts)
chore(deps): update convict
chore(deps): update morgan
chore(deps): update eslint-plugin-jest
chore(deps): update babel-eslint
chore(deps): removed nodemon
chore(deps): update @types/convict
chore(deps): set @types/jest to match jest version
chore(deps): set @babel/core to ^7.9.0
chore(deps): set @babel/preset-env to 7.9.0
chore(deps): update hapi
In order to keep the typescript shared modules up to date for their dependents to use while running locally this adds them to pm2 so that a `tsc --watch` can run alongside the other services.
Note, this DOES NOT automatically restart dependent javascript service processes.
This change refactors pm2-all.sh in order to start processes in dependency topological order. This requires an update to most pm2.config.js files in order to set PATH correctly while running under yarn.
This reduces the boilerplate when all your package
needs is to copy the contents of it's own directory to /app
and a pm2.config.js file exists
To allow more packages to use the default, /fxa-geodb and
/fxa-shared are included in the fxa-node base image.
Custom images can still be made by having a Dockerfile as before.
This adds a base node image and builder image so
that all our service images can share the same
common base, be smaller, and require less customization.
why, after all these years?
because there's a few annoying cases where the loopback
makes dev harder. When you try to test a flow from a not
local machine, like a tv, mobile device, or a saucelabs
proxy session. With localhost it's easy enough to forward
but loopback not so much
enough is enough
why: several reasons...
- to have a unified approach and pattern for:
- debugging
- fs watching for local dev
- running services in dev
- configuring services in dev
- to improve the initial clone and subsequent `npm ci` experience
- to make future work on tooling easier
It's pretty disheartening when your PR fails in ci because of an
npm advisory unrelated to your changes. We still need to be aware
of them but have the ability to more easily keep doing other work.
Running the audit on git push allows us all to see advisories
quickly, but allows us to override and push with --no-verify so
work can continue and the advisory addressed on its own.
dependabot[bot]
dschom
Peter deHaan
Danny Coates
johngruen
Jared Hirsch
Jody Heavener
Dave Justice
Ben Bangert
Lauren Zugai