This makes sure that serving the documentation via poetry does not regress
and we have a single way of generating them.
Setting the build backend in `pyproject.toml` makes `pip` to use `poetry` to install the dependencies. By doing so, readthedocs will be able to install the required dependencies to generate the docs (since they don't support poetry directly).
Work to support running Python tests outside of the Docker container.
This also helps with integration with IDEs and is prework for using Git hooks for precommit/prepush features.
* Change permissions for docker/entrypoint.sh
* Allow using Django in localhost
* Documentation changes
Allow running shellcheck tests
Use docker-compose for the Selenium tests
Install docs updates
Switch to yarn cache instead of npm
Install UI dependencies
* Add ability to override DATABASE_URL and use a .env file
So that their purposes are clearer and it's easier to differentiate
between generated content and files committed to the repository.
* Neutrino build: `build/` -> `.build/`
* Sphinx build: `_build/` -> `.build-docs/`
* Django collectstatic: `treeherder/static/` -> `.django-static/`
Neutrino controls our frontend linting, transpilation, source-maps,
testing, dev-server and optimisation of production builds.
Highlights of the upgrade are:
* Major version updates to the individual tools within (such as webpack,
Babel and ESLint), significantly improving performance, fixing
transpilation/minification correctness bugs, adding support for newer
ECMAScript features, and increasing linter coverage.
* Hot reloading in the dev server now works for all entry-points and not
just the jobs view, shortening the feedback cycle.
* Reduced bundle size due to webpack 4's tree shaking, scope hoisting,
automatic shared/vendor code chunk splitting (no need for the manually
maintained 'vendor' list).
* CSS is now extracted out of JS, which improves performance, reduces
bundle size and prevents the initial white flash of un-styled content.
* Support for dynamic imports/code splitting (needed for bug 1502192).
* Support for Jest via a new Jest preset (unblocks bug 1364045).
* Support for public class field declarations (unblocks bug 1480166).
* Improved source-maps (increases the quality of production exception
trace-backs and fixes several debugger breakpoint bugs).
* Reduced amount of custom configuration required for our fairly complex
frontend needs, reducing maintenance burden and allowing for easier
future Neutrino upgrades.
In addition this PR:
* Fixes the WhiteNoise `immutable_file_test()` regex, so that it now
correctly enables browser caching of images, fonts and source maps.
* Enables webpack-dev-server's overlay feature, which displays any
compilation errors in the browser, saving having to switch back
to the console (this can be enabled for warnings too if desired).
* Enables webpack-dev-server's automatic browser-opening feature,
which saves having to manually navigate to `localhost:5000` after
running `yarn start`.
* Switches Karma tests to run Firefox in headless mode, reducing the
workflow disruption when running `yarn test`.
* Uses the new webpack `performance` option to enable maximum asset
file size thresholds, to help prevent bundle-size regressions.
* Rewrites the `package.json` script commands so that they now work
correctly on Windows, even when setting environment variables.
Performance comparison:
* Local `yarn build`:
- Cached: 2m34s -> 23s
- Uncached: 2m34s -> 58s
* Local `yarn start`:
- Cached: 34.5s -> 13.6s
- Uncached: 34.5s -> 31.3s
* Local `yarn test`
- Cached: 61.5s -> 19.8s
- Uncached: 61.5s -> 22.0s
* Local `yarn lint`
- Cached: 3.8s -> 1.8s
- Uncached: 13.7s -> 13.4s
* Travis end-to-end time:
9 minutes -> 6 minutes
* Heroku deploy end-to-end time:
14 minutes -> 9 minutes
Enabling the ESLint caching reduces `yarn lint` time by 60%. See:
https://eslint.org/docs/user-guide/command-line-interface#caching
This also adds JS files in the repo root to the list of files that
are linted, and fixes a subsequent lint error in `.eslintrc.js`.
The new log viewer loads the logs directly in the client from wherever
they are stored, so doesn't need Treeherder's API to proxy them.
The logslice API was the only user of the Django filesystem cache, so
that has also been removed.
Since it is footgun-prone, discourages upstreaming of useful development
tricks & is unnecessary in an environment variable centric world.
The one remaining `BZ_API_URL` setting isn't actively used, and if this
changes in the future, it should be set via an environment variable
instead.
Since they are deprecated and all submitters have switched over to using
Hawk credentials instead.
The automatically created migrations file was edited to remove the
unused `models` import, since otherwise flake8 complains. We could
alternatively exclude the migrations directory from flake8, however we
would then miss linter errors in any hand-written migrations files.
In addition, Django have fixed the issue in 1.9:
a7bc00e17b
In update.py, the line outputting revision.txt has to be moved later,
since the `dist/` directory won't exist until grunt build has run. In
addition, since `grunt build` removes the entire `dist/` directory, we
no longer need to manually remove *.gz.
We use the `--production` options for both `npm install` and
`grunt build`, so that the `devDependencies` in package.json are
ignored, and we only install/load the ones listed under `dependencies`
in package.json - since that's all that is required for the build.
We have to use `./node_modules/.bin/grunt` rather than `grunt`, since
grunt-cli is not installed globally on the treeherder admin machine for
greater isolation between stage and production.
Since they're not specific to the Django app 'webapp'.
Whilst we're there, the local & example settings files have been
renamed. In the future I'd like to combine settings_local.example.py
with puppet/files/treeherder/local.vagrant.py, but I'll do that in
another bug.
On Heroku, there is no load balancer or Varnish-like cache in front of
gunicorn, so we must handle gzipping responses in the app.
In order for WhiteNoise to serve gzipped static content, assets must be
gzipped on disk in advance (doing so on-demand in Python would not be
as performant). WhiteNoise will then serve the `.gz` version of files in
preference to the original, if the client indicated it supported gzip.
For assets covered by Django's collectstatic, gzipping the assets only
requires using WhiteNoise's GzipManifestStaticFilesStorage backend,
which wraps Django's ManifestStaticFilesStorage to create hashed+gzipped
versions of static assets:
http://whitenoise.evans.io/en/latest/django.html#add-gzip-and-caching-support
The collectstatic generated files will then contain the file hash in
their filename, so WhiteNoise can also serve them with a large max-age
to avoid further requests if the file contents have not changed.
For the UI files under `dist/`, we cannot rely on the Django storage
backend, since the directory isn't covered by STATICFILES_DIRS (it is
instead made known to WhiteNoise via `WHITENOISE_ROOT`). As such, files
under `dist/` are gzipped via an additional step during deployment. See:
http://whitenoise.evans.io/en/latest/base.html#gzip-support
Files whose extension is on the blacklist, or that are not >5% smaller
when compressed, are skipped during compression.
Since WhiteNoise won't serve files from /media/, so the existing file
in `treeherder/webapp/media/` 404s when accessed via:
https://treeherder.{mozilla,allizom}.org/media/revision
IMO the site root makes more sense for this file anyway, so let's just
save it under `dist/`. Also adds a .txt extension for clarity.
The old file has been left for now to ease the transition, and will be
deleted once the IRC pushbots config and What's Deployed URLs have been
updated.
A json file is overkill, and it is currently in a directory intended for
production-specific files, so move the blacklist inline. We should
probably move the whole analyser to another file, but that can be done
later.
Also store the analyser output in the root of the media directory, since
a subdirectory is unnecessary. The media directory is now empty, so we
must use .gitkeep to ensure it is created.
ionutgoldan
Armen Zambrano
Shubhank Saxena
Kyle Lahnakoski
Armen Zambrano G
Cameron Dawson
Ed Morley
pyup.io bot
Ed Morley
Casey Williams
Ed Morley
Kris Rivera
James Graham
Mauro Doglio
pnispel
Jonathan Eads