This makes sure that serving the documentation via poetry does not regress
and we have a single way of generating them.
Setting the build backend in `pyproject.toml` makes `pip` to use `poetry` to install the dependencies. By doing so, readthedocs will be able to install the required dependencies to generate the docs (since they don't support poetry directly).
* Add Travis job to run Python tests outside of Docker
* `runtests.sh` is renamed to `runchecks.sh` and it does not run Python tests
* `manage.py check --deploy` was duplicated in Travis
* Update testing documentation
* Remove `-bb` since it is not needed since Python 3.5
This increases the chances of not having a Heroku build fail and catch it in Travis instead.
It also cleans up a bit the file and removes incorrect usage of `env` values.
If any step on the Travis run fails (like formatting checks or a test) then
we don't report the code coverage for it.
Unfortunately, this can be distracting when we see status checks
reporting a failure or a big change.
This change makes code coverage to report even if there are minor
failures.
Sometimes Travis fails because `nvm` would not be able to reach the node package with:
> Failed to install 12. Remote repository may not be reachable.
Perhaps being more specific will help with timing of node releases or more likely hitting the cache.
Up until now we were not testing the same code path that Heroku follows to
generate the front-end build.
Heroku runs the `yarn install` command under `NODE_ENV=production` which causes
only the production dependencies to be installed. In the case of
[PR 5283](#5283), we moved `neutrino`
from `dependencies` to `devDependencies` and went unnoticed until Heroku tried to deploy
the change. See output [1].
This change adds one more entry for Travis where we only install the production dependencies
before running `yarn build`.
[1]
```shell
2019-08-16 19:34:49 -----> Build
2019-08-16 19:34:49 Detected both "build" and "heroku-postbuild" scripts
2019-08-16 19:34:49 Running heroku-postbuild (yarn)
2019-08-16 19:34:50 yarn run v1.17.3
2019-08-16 19:34:50 $ yarn build
2019-08-16 19:34:50 $ node ./node_modules/webpack/bin/webpack.js --mode production
/tmp/build_375161ca12e2f192c2c9b1caab8276cd/node_modules/webpack-cli/bin/cli.js:93
throw err;
^
Error: Cannot find module 'neutrino'
```
Up until now we were not testing the same code path that Heroku follows to
generate the front-end build.
Heroku runs the `yarn install` command under `NODE_ENV=production` which causes
only the production dependencies to be installed. In the case of
[PR 5283](https://github.com/mozilla/treeherder/pull/5283), we moved `neutrino`
from `dependencies` to `devDependencies` and went unnoticed until Heroku tried to deploy
the change. See output [1].
This change adds one more entry for Travis where we only install the production dependencies
before running `yarn build`.
[1]
```shell
2019-08-16 19:34:49 -----> Build
2019-08-16 19:34:49 Detected both "build" and "heroku-postbuild" scripts
2019-08-16 19:34:49 Running heroku-postbuild (yarn)
2019-08-16 19:34:50 yarn run v1.17.3
2019-08-16 19:34:50 $ yarn build
2019-08-16 19:34:50 $ node ./node_modules/webpack/bin/webpack.js --mode production
/tmp/build_375161ca12e2f192c2c9b1caab8276cd/node_modules/webpack-cli/bin/cli.js:93
throw err;
^
Error: Cannot find module 'neutrino'
```
* Change permissions for docker/entrypoint.sh
* Allow using Django in localhost
* Documentation changes
Allow running shellcheck tests
Use docker-compose for the Selenium tests
Install docs updates
Switch to yarn cache instead of npm
Install UI dependencies
* Add ability to override DATABASE_URL and use a .env file
- Adds a new "infrastructure" section to the docs, which describes
architecture, administration and troubleshooting (fixes bug 1165259).
- Adds code comments to any deployment-related files in the repository.
- Adds documentation for the various ways in which users can access
Treeherder data (fixes bug 1335172).
- Reorganises the structure of some of the existing non-infrastructure
docs, to make the documentation easier to navigate.
This makes Heroku/Vagrant use Python 3.6.8 instead of Python 2.7, and
inverts the versions used in the Travis testing matrix - leaving all
unit tests running against Python 2.7 to make it easier to roll back
if needs be. The Vagrant provision script and Heroku Python buildpack will
automatically detect the changed Python version and purge the existing
installation/site-packages prior to installing the new version.
We're using Python 3.6 rather than 3.7, since latest Celery/Kombu do not
yet support Python 3.7 (and we're on an older version anyway; bug 1337717).
Bug 1529243 is filed for updating to Python 3.7 later.
Also remove all Karma support and update the docs to only mention ``Jest``.
One of the test files was testing some AngularJS filters. I converted these
tests to test the equivalent helper functions.
Switches from Sphinx to MkDocs, since it:
* supports Markdown natively without requiring hacks like `eval_rst`
* validates inline links, ensuring that they are not broken
* has a more pleasant live-reloading dev-server
* supports the nicer looking mkdocs-material theme
* is a third of the size of Sphinx (including deps)
The theme change is now possible since Read the Docs have just started
supporting use of custom themes (previously they would override the theme
and use `readthedocs` theme regardless).
Currently Prettier is used via the ESLint integration, which only supports
JS/JSX and not the other filetypes Prettier is able to format.
For now these additional filetypes are excluded by `.prettierignore`
entries, however these will be removed/fixes applied in later commits.
Neutrino controls our frontend linting, transpilation, source-maps,
testing, dev-server and optimisation of production builds.
Highlights of the upgrade are:
* Major version updates to the individual tools within (such as webpack,
Babel and ESLint), significantly improving performance, fixing
transpilation/minification correctness bugs, adding support for newer
ECMAScript features, and increasing linter coverage.
* Hot reloading in the dev server now works for all entry-points and not
just the jobs view, shortening the feedback cycle.
* Reduced bundle size due to webpack 4's tree shaking, scope hoisting,
automatic shared/vendor code chunk splitting (no need for the manually
maintained 'vendor' list).
* CSS is now extracted out of JS, which improves performance, reduces
bundle size and prevents the initial white flash of un-styled content.
* Support for dynamic imports/code splitting (needed for bug 1502192).
* Support for Jest via a new Jest preset (unblocks bug 1364045).
* Support for public class field declarations (unblocks bug 1480166).
* Improved source-maps (increases the quality of production exception
trace-backs and fixes several debugger breakpoint bugs).
* Reduced amount of custom configuration required for our fairly complex
frontend needs, reducing maintenance burden and allowing for easier
future Neutrino upgrades.
In addition this PR:
* Fixes the WhiteNoise `immutable_file_test()` regex, so that it now
correctly enables browser caching of images, fonts and source maps.
* Enables webpack-dev-server's overlay feature, which displays any
compilation errors in the browser, saving having to switch back
to the console (this can be enabled for warnings too if desired).
* Enables webpack-dev-server's automatic browser-opening feature,
which saves having to manually navigate to `localhost:5000` after
running `yarn start`.
* Switches Karma tests to run Firefox in headless mode, reducing the
workflow disruption when running `yarn test`.
* Uses the new webpack `performance` option to enable maximum asset
file size thresholds, to help prevent bundle-size regressions.
* Rewrites the `package.json` script commands so that they now work
correctly on Windows, even when setting environment variables.
Performance comparison:
* Local `yarn build`:
- Cached: 2m34s -> 23s
- Uncached: 2m34s -> 58s
* Local `yarn start`:
- Cached: 34.5s -> 13.6s
- Uncached: 34.5s -> 31.3s
* Local `yarn test`
- Cached: 61.5s -> 19.8s
- Uncached: 61.5s -> 22.0s
* Local `yarn lint`
- Cached: 3.8s -> 1.8s
- Uncached: 13.7s -> 13.4s
* Travis end-to-end time:
9 minutes -> 6 minutes
* Heroku deploy end-to-end time:
14 minutes -> 9 minutes
The Python 3 sub-job runs more than just linters, so the old name
was not correct. The comments have been removed since they mostly
duplicate the name in `env`.
* Adds a `rabbitmq-server` install step, since it's no longer
installed in the image by default.
* Removes the MySQL 5.7 install step, since Ubuntu 16.04's default
MySQL is now 5.7 rather than 5.6.
* Adjusts which services are manually started to account for changes
in which are enabled by default in the new image.
This enables `DeprecationWarnings` for things that Python 2 knows
are not compatible with Python 3. The `error` entry in the pytest
`filterwarnings` setting ensures these will be surfaced as Exceptions
and so result in test failures.
See:
https://docs.python.org/2/using/cmdline.html#cmdoption-3
The removal of `sorted()` from `test_bug_job_map_api.py` is to fix:
`DeprecationWarning: dict inequality comparisons not supported in 3.x`
This merges the three separate non-selenium Python test jobs into
one, reducing the overall job count from 7 to 5. This helps avoid
hitting Travis concurrency limits, that delay starting the selenium
job (which is the long pole for the end to end time).
Armen Zambrano
Cameron Dawson
Shubhank Saxena
Shubham Kumar
Sarah Clements
Kyle Lahnakoski
Armen Zambrano G
Renovate Bot
Sarah Clements
Ed Morley
George Hickman
Dave Hunt