Work to support running Python tests outside of the Docker container.
This also helps with integration with IDEs and is prework for using Git hooks for precommit/prepush features.
* Change permissions for docker/entrypoint.sh
* Allow using Django in localhost
* Documentation changes
Allow running shellcheck tests
Use docker-compose for the Selenium tests
Install docs updates
Switch to yarn cache instead of npm
Install UI dependencies
* Add ability to override DATABASE_URL and use a .env file
The id token payload contains an `exp` property, which is an integer
representing the number of seconds past the epoch at which the id token
expires.
However the mocked value in our authentication tests was the string `'500'`,
which is neither the correct data type, nor a timestamp. This meant that
during tests only, the `min(accesstoken_exp_in_ms, idtoken_exp_in_ms)`
in `AuthBackend.authenticate()` was comparing an int and a string, which
under Python 3 results in:
`TypeError: '<' not supported between instances of 'str' and 'int'`
A later bug/PR will refactor the auth backend to fix issues unrelated to
Python 3 compatibility and add more test coverage.
This makes Heroku/Vagrant use Python 3.6.8 instead of Python 2.7, and
inverts the versions used in the Travis testing matrix - leaving all
unit tests running against Python 2.7 to make it easier to roll back
if needs be. The Vagrant provision script and Heroku Python buildpack will
automatically detect the changed Python version and purge the existing
installation/site-packages prior to installing the new version.
We're using Python 3.6 rather than 3.7, since latest Celery/Kombu do not
yet support Python 3.7 (and we're on an older version anyway; bug 1337717).
Bug 1529243 is filed for updating to Python 3.7 later.
So that their purposes are clearer and it's easier to differentiate
between generated content and files committed to the repository.
* Neutrino build: `build/` -> `.build/`
* Sphinx build: `_build/` -> `.build-docs/`
* Django collectstatic: `treeherder/static/` -> `.django-static/`
Neutrino controls our frontend linting, transpilation, source-maps,
testing, dev-server and optimisation of production builds.
Highlights of the upgrade are:
* Major version updates to the individual tools within (such as webpack,
Babel and ESLint), significantly improving performance, fixing
transpilation/minification correctness bugs, adding support for newer
ECMAScript features, and increasing linter coverage.
* Hot reloading in the dev server now works for all entry-points and not
just the jobs view, shortening the feedback cycle.
* Reduced bundle size due to webpack 4's tree shaking, scope hoisting,
automatic shared/vendor code chunk splitting (no need for the manually
maintained 'vendor' list).
* CSS is now extracted out of JS, which improves performance, reduces
bundle size and prevents the initial white flash of un-styled content.
* Support for dynamic imports/code splitting (needed for bug 1502192).
* Support for Jest via a new Jest preset (unblocks bug 1364045).
* Support for public class field declarations (unblocks bug 1480166).
* Improved source-maps (increases the quality of production exception
trace-backs and fixes several debugger breakpoint bugs).
* Reduced amount of custom configuration required for our fairly complex
frontend needs, reducing maintenance burden and allowing for easier
future Neutrino upgrades.
In addition this PR:
* Fixes the WhiteNoise `immutable_file_test()` regex, so that it now
correctly enables browser caching of images, fonts and source maps.
* Enables webpack-dev-server's overlay feature, which displays any
compilation errors in the browser, saving having to switch back
to the console (this can be enabled for warnings too if desired).
* Enables webpack-dev-server's automatic browser-opening feature,
which saves having to manually navigate to `localhost:5000` after
running `yarn start`.
* Switches Karma tests to run Firefox in headless mode, reducing the
workflow disruption when running `yarn test`.
* Uses the new webpack `performance` option to enable maximum asset
file size thresholds, to help prevent bundle-size regressions.
* Rewrites the `package.json` script commands so that they now work
correctly on Windows, even when setting environment variables.
Performance comparison:
* Local `yarn build`:
- Cached: 2m34s -> 23s
- Uncached: 2m34s -> 58s
* Local `yarn start`:
- Cached: 34.5s -> 13.6s
- Uncached: 34.5s -> 31.3s
* Local `yarn test`
- Cached: 61.5s -> 19.8s
- Uncached: 61.5s -> 22.0s
* Local `yarn lint`
- Cached: 3.8s -> 1.8s
- Uncached: 13.7s -> 13.4s
* Travis end-to-end time:
9 minutes -> 6 minutes
* Heroku deploy end-to-end time:
14 minutes -> 9 minutes
* Enables the display of skipped test/expected fail reasons, in
the pytest summary.
* Skips the Selenium tests with a clear reason message, unless the
built UI is found (preventing the annoying/confusing test timeouts).
* Removes the disabling of the `pytest-html` and `pytest-metadata`
plugins, since they are required when passing the `--html` option
to generate an HTML report.
* Updates the docs to mention `yarn build` and `--html`.
* Switches from the `ignore` setting to the new `extend_ignore`, which
doesn't overwrite the default ignore list, meaning we no longer have
to duplicate it ourselves.
* Remove the rarely used `[pycodestyle]` config section, since it's
only used when using tools like autopep8, which should really learn
to use the `[flake8]` section themselves.
* Enables the previously ignored F403 and F405 rules, adding `# noqa`
entries to instances that we do not wish to fix.
* Adjust max line length down to 100, since we already disable the
`E501: line too long` rule, making the length mostly redundant
other than in IDEs, where it's probably good to show a warning if
exceeding 100 characters.
* Fixes:
```
treeherder/intermittents_commenter/commenter.py:202:10:
W605 invalid escape sequence '\['
treeherder/intermittents_commenter/commenter.py:202:24:
W605 invalid escape sequence '\]'
treeherder/webapp/graphql/schema.py:7:1:
F403 'from treeherder.model.models import *' used; unable to detect undefined names
```
Closes#4177.
Refs #3425.
Refs #3565.
* Renames `is_immutable_file` to match changes to the parent class.
* Suppresses the new "missing static files directory" warning when
running the tests (since it's expected in !production).
* Uses the new `extras_require` `[brotli]` entry to make WhiteNoise's
dependency more obvious.
Changelog:
http://whitenoise.evans.io/en/stable/changelog.html#v4-0Closes#3903.
Since Read The Docs has a pre-installed older version installed,
and doesn't use `--upgrade` with pip, so we have to force the latest
version to be installed by using "editable" (`-e`) mode:
https://pip.pypa.io/en/stable/reference/pip_install/#vcs-support
This fixes the exception that occurred on Read the Docs (but wasn't
seen on Travis, since it doesn't have the pre-installed older version).
This enables `DeprecationWarnings` for things that Python 2 knows
are not compatible with Python 3. The `error` entry in the pytest
`filterwarnings` setting ensures these will be surfaced as Exceptions
and so result in test failures.
See:
https://docs.python.org/2/using/cmdline.html#cmdoption-3
The removal of `sorted()` from `test_bug_job_map_api.py` is to fix:
`DeprecationWarning: dict inequality comparisons not supported in 3.x`
This stops the broken mozlog plugin being loaded, which was preventing
us updating to newer pytest. Other unused plugins have also been
disabled, to avoid future issues and improve pytest load times.
Before this change running pytest with no arguments took 51 seconds
to just collect the list of tests to run (`pytest --collect-only`),
since the entire source directory was scanned (including the 18,000
files in `node_modules`).
Now it takes only 3.7 seconds.
Initially these will continue to be run in Jenkins, however later they
will be converted to run on Travis along with the existing local
Selenium tests.
Since it's actually needed to catch unused imports, albeit it's a bit
more strict than it used to be, so required some fix-ups before it could
be re-enabled.
Now that we're on Django 1.10 (which doesn't add unused imports) and the
old migrations have been removed we no longer need to ignore them and
risk hiding real breakage.
treeherder/config/wsgi.py:17:1: E402 module level import not at top of file
treeherder/config/wsgi.py:18:1: E402 module level import not at top of file
These forbid common constructs, notably
if (foo and
bar):
which has no basis in PEP8 or sanity. In modern pyflakes only E129 would have to be permitted
but Ubuntu 14.04 has an older version which lumps this in with some other cases in E125.
The MPL 2.0 terms state that as long as a LICENSE file is present, the
per-file header text is not required. See "Exhibit A" at the end of:
https://www.mozilla.org/MPL/2.0/
Since it only speeds up parsing by a few percent of total runtime, and
is therefore not worth the added complexity for deployment and local
hack-test-debug cycles when working on the log parser.
The .gitignore and update.py entries will be removed in a later commit,
once the stage/prod src directories have been cleaned up.
Kyle Lahnakoski
Armen Zambrano
Ed Morley
dependabot[bot]
Cameron Dawson
pyup.io bot
George Hickman
Dave Hunt
Ed Morley
Rebecca Billings
Ed Morley
James Graham
William Lachance