With the changes in previous commits, all of the assets that were
originally manually copied to `dist/img/` are now correctly handled
by webpack as dependencies (and so emitted to `dist/` automatically).
As such, this leaves only three files that need copying from `src/`,
so they are now listed explicitly to avoid having to continually
update `ignore` to prevent extra files from sneaking in:
https://webpack.js.org/plugins/copy-webpack-plugin/
As a result of this change, the following assets are no longer
needlessly created under `dist/` as part of `yarn build`:
```
img/dancing_cat.gif
img/line_chart.png
img/logviewerIcon.png
img/logviewerIcon.svg
img/logviewerIconHelp.svg
img/tip.png
img/tip-locked.png
img/tree.xcf
img/tree_closed.png
img/tree_open.png
img/treeherder-logo.png
```
To confirm that this would not break anything, the JS and HTML files
under `dist/` were grepped for the string `img/`, and there are no
references remaining.
Currently `revision.txt` only exists on Heroku, since it's generated
by the Heroku-only `post_compile` script, just prior to `yarn build`.
However this means:
* HTTP 404s of `revision.txt` are seen in the browser console when
developing locally, which gives the appearance of something being
broken, even though it's not.
* when we convert the wildcard `CopyPlugin` rule to an explicit list
of files to copy (in a later commit), it will cause errors when
building locally, since `CopyPlugin` expects all declared files
to exist.
Adding a placeholder file prevents both of the above.
This ensures that webpack knows they are a dependency, meaning:
* no need to manually copy them to `dist/img/` using `CopyPlugin`
(the wildcard copy rules will be cleaned up in a later commit)
* they are inlined as a base64 encoded data URI by `url-loader`.
The changes to `thFaviconLink` are required to prevent:
```
Error: [$interpolate:interr] Can't interpolate: {{favicon}}
Error: [$sce:insecurl] Blocked loading resource from url not allowed
by $sceDelegate policy. URL: data:image/png;base64,...
```
Which are due to AngularJS not trusting data URIs by default. See:
https://docs.angularjs.org/error/$sce/insecurl
Previously `html-loader` only parsed `<img src="...">` tags when
looking for assets/dependencies. Now the `<link href="...">` tags
for favicons are processed too, which means `img/tree_open.png`
and friends will be included in the webpack build and not need to
be manually copied into `dist/img/`:
https://webpack.js.org/loaders/html-loader/
This does not visible change the number of hashed images output to
`dist/`, since the favicons are small enough that `url-loader` inlines
them in the HTML as base64 encoded data URIs (this is adjustable if
not desired later):
https://webpack.js.org/loaders/url-loader/
Neutrino 4 configures `file-loader` as the loader for HTML (rather than
the more usual `html-loader`), which means the HTML is not parsed to
look for further dependencies such as `<img src="...">` tags. Our
custom Neutrino config overrode that to `raw-loader` (presumably to
work around bugs caused by the use of `file-loader`), which doesn't
parse HTML either.
Instead, these assets were being manually copied to `dist/img/` by
`neutrino-custom/production.js`'s `CopyPlugin` rule, effectively
circumventing the webpack build process.
Newer Neutrino correctly uses `html-loader`, causing our HTML to be
parsed during the webpack build for the first time. However now that
the images are being resolved at build time rather than runtime, the
relative paths need to be updated to account for the directory layout
differences between `src/` and `dist/`, to prevent build errors.
A significant benefit of this change is that images referenced from
HTML will now be output with hashed filenames, meaning they get given
long-lived `Cache-Control` headers by WhiteNoise.
See:
https://webpack.js.org/loaders/file-loader/https://webpack.js.org/loaders/raw-loader/https://webpack.js.org/loaders/html-loader/
Neutrino 4 configured the SVG mimetype as `application/svg+xml`, which
Firefox doesn't appear to like. Neutrino 8 instead doesn't specify a
mimetype directly, allowing `url-loader` to autodetect via the NPM
`mime` package (which returns `image/svg+xml` for SVGs).
This switches to the Neutrino 8 approach, to prevent the job detail
panel's log viewer icon breaking in later commits when we start
parsing HTML properly (which causes url-loader to embed the icon as
a base64 encoded data URI rather than a URL to a file).
Neutrino 4 only configured `url-loader` for use with `.png` and `.jpg`.
This means that once we start parsing HTML properly in later commits,
we would otherwise get the following error:
```
ERROR in ./ui/img/dancing_cat.gif
Module parse failed: C:\Users\Ed\src\treeherder\ui\img\dancing_cat.gif Unexpected character '�' (1:6)
You may need an appropriate loader to handle this file type.
(Source code omitted for this binary file)
@ ./ui/partials/perf/comparectrl.html 1:214-250
@ ./ui/partials \.html$
@ ./ui/js/cache-templates.js
@ ./ui/js/perfapp.js
@ ./ui/entry-perf.js
@ multi ./ui/entry-perf.js
```
Presently, when the build system of a job is buildbot, there is a hyper
link on the machine name that takes a user to the slave health
dashboard. When a job runs outside of buildbot, there is a worker
explorer. A worker explorer shows information like most recent task IDs
claimed. Also, a user can perform actions on the machine if they own the
required scopes (e.g., quarantine & terminate instance). This commit
provides a shortcut to the worker explorer when the build system is not
buildbot.
Since memcached server and the client headers are now unused.
Whilst the `zlib1g-dev` package is no longer needed by pylibmc, it's
still required as a sub-dependency of `libmysqlclient-dev`, so has
deliberately not been added to the `apt-get purge` cleanup step.
Since:
* Redis has additional features we need (eg for bug 1409679)
* the Redis server, python client and Django backend are more
actively maintained (so have persistent connections/pooling that
actually works, which gives a big perf win)
* we can use Heroku's own Redis addon rather than relying on a
third-party's (to hopefully prevent a repeat of the certificate
expiration downtime)
This commit:
* Switches pylibmc/django-pylibmc to redis-py/django-redis, and
configures the new backend according to:
http://niwinz.github.io/django-redis/latest/https://github.com/andymccurdy/redis-py
* Uses redis-py's native support for TLS to connect to the Heroku
Redis server's stunnel port directly, avoiding the complexity of
using a buildpack to create an stunnel between the dyno and server:
https://devcenter.heroku.com/articles/securing-heroku-redis#connecting-directly-to-stunnel
* Uses explicit `REDIS_URL` values on Travis/Vagrant rather than
relying on the django-environ `default` value (for parity with
how we configure `DATABASE_URL` and others).
* Removes the pylibmc connection-closing workaround from `wsgi.py`.
Note: Whilst the Heroku docs suggest using `django-redis-cache`, it's
not as actively maintained, so we're using `django-redis` instead:
https://devcenter.heroku.com/articles/heroku-redishttps://github.com/niwinz/django-redishttps://github.com/sebleier/django-redis-cache
Before this is merged, Heroku Redis instances will need to be created
for stage/production (prototype done) - likely on plan `premium-3`:
https://elements.heroku.com/addons/heroku-redis
We'll also need to pick an eviction policy:
https://devcenter.heroku.com/articles/heroku-redis#maxmemory-policy
Once deployed, the `memcachier-tls-buildpack` buildpack will no longer
be required and can be removed from prototype/stage/prod, along with
the `TREEHERDER_MEMCACHED` environment variable and Memcachier addon.
Previously the `cache.set()` calls made by buildapi and pushlog
ingestion did not pass a timeout argument, so used the `settings.py`
global default of `None` (which means never expire).
There are a few disadvantages with this approach:
1) The `cache.set()` calls can't be fully interpreted on their own.
2) Never expiring means stale keys could be left behind (for example
the last push ID key, when a repository is removed).
3) If Django core functionality or third-party packages use the cache
but don't set their own timeout, they may be expecting the Django
default timeout of 5 minutes (or at least a finite timeout).
Now all cache writes in Treeherder explicitly set a timeout, and the
global timeout is restored to the Django default of 5 minutes:
https://docs.djangoproject.com/en/1.11/ref/settings/#timeout
Uses our custom `clear_cache` Django management command instead of
restarting memcached, since (a) for memcached has the same effect,
(b) will continue to work even after we switch to Redis (which instead
persists cache contents across server restarts).
Removes the `threstartmemcached` command since IMO it's no simpler
than just running `./manage.py clear_cache` on its own.
* Adjusts the test name and docstring, since there is nothing
memcached-specific about the test.
* Overrides the default timeout of never to 10 seconds, to prevent
previous test run values persisting.
Jonathan French
pyup.io bot
Ed Morley
Cameron Dawson
Rob Wood
Vincenzo Scalzi
Hassan Ali
renovate[bot]
Dave Hunt
Kartikaya Gupta