Граф коммитов

202 Коммитов

Автор SHA1 Сообщение Дата
Sarah Clements 7ea0760c58
Remove heroku v2 (#7271)
* update dockerfile and remove post_compile
* remove more heroku mentions
* remove redis heroku ref
* update middleware and CSP
* remove irrelevant test and script references to heroku
remove tox heroku command from circleCI config
* more references removed
* remove prototype2 reference
* remove old TLS check for redis
2021-09-16 11:30:12 +01:00
Sarah Clements d7849ac7af
Revert "heroku cleanup (#7219)" (#7245)
This reverts commit 7cb8d8abe7.
2021-08-18 18:22:13 +01:00
Sarah Clements 7cb8d8abe7
heroku cleanup (#7219)
* update dockerfile and remove post_compile
* remove redis heroku ref
* update middleware and CSP
* remove irrelevant test and script references to heroku
remove tox heroku command from circleCI config
* remove get_tls_redis_url and reconfigure default TLS_PATH_CERT in docker file
2021-08-16 14:49:07 +01:00
Ben Hearsum (he/him) 499889defc
Drop newrelic record-deploy, because it depends too heavily on heroku variables (#7147) 2021-05-21 09:44:00 -04:00
Sarah Clements 154cb8c59d
Change pre_deploy to remove django checks (#7083)
* For some reason the django check warning is causing a release to fail for a recent schema change
so removing the check since its implicitly called as part of the migrate command that follows
2021-04-01 13:30:21 -07:00
Armen Zambrano G f00afa8e1e
Unify linting (local pre-commit hooks & CI) (#6993)
* Prevent regression of pre-commit checks
* Switch to new prettier repo
2021-01-29 15:47:06 -08:00
Cameron Dawson 779b58fb73
Fix New Relic after upgrading python package (#6764) 2020-09-16 13:46:20 -07:00
Armen Zambrano 13c1fc851d
Bug 1626914 - Make Review Apps use Pulse credentials secret (#6241)
* Bug 1626914 - Make Review Apps use Pulse credentials secret

In the beginning of the Heroku Review Apps it made sense to create a user/password that was shared accross apps and be publicly available (since each app would generate their own random queue names).

Unfortunately, last night, I saw a lot of alerts for queues that were overgrowing. We recently
landed some code that should be deleting queues once there are no consumers (which was not happening). This could be because we had a Review app from a long time ago create lots of queues that were not auto-deleted or someone locally is doing development with the shared credentials unknowingly creating many queues.

Either way, the new Heroku Review apps allow defining config variables (secrets) at the pipeline level and are injected in the Review apps upon creation.

I've reset the treeherder-shared-pulse-user password and created a PULSE_URL config var with the new password.

* Checking for NEW_RELIC_INSIGHTS_API_KEY prevents review apps from releasing
2020-04-03 10:28:48 -04:00
Cameron Dawson 61ebbb30e8
Bug 1612229 - Push Health Usage Dashboard (#6137) 2020-03-16 14:25:11 -07:00
Armen Zambrano 66f8f5d344
Travis: Add Heroku's Python build execution path (#6057)
This increases the chances of not having a Heroku build fail and catch it in Travis instead.

It also cleans up a bit the file and removes incorrect usage of `env` values.
2020-03-10 08:20:24 -04:00
Armen Zambrano G 725700a8a2 Do not run newrelic-admin in release phase if running in preview app 2019-11-12 09:09:10 -05:00
Cameron Dawson c94d8fbadf Fix New Relic record-deploy parameters 2019-10-03 13:03:20 -07:00
Armen Zambrano deaf8c850c Bug 1169263 - Switch from Vagrant to docker-compose (#4901)
* Change permissions for docker/entrypoint.sh
* Allow using Django in localhost
* Documentation changes
Allow running shellcheck tests
Use docker-compose for the Selenium tests
Install docs updates
Switch to yarn cache instead of npm
Install UI dependencies
* Add ability to override DATABASE_URL and use a .env file
2019-05-16 14:56:02 -07:00
Ed Morley 6945c4c471
Bug 1165259 - Add infrastructure documentation (#4766)
- Adds a new "infrastructure" section to the docs, which describes
  architecture, administration and troubleshooting (fixes bug 1165259).
- Adds code comments to any deployment-related files in the repository.
- Adds documentation for the various ways in which users can access
  Treeherder data (fixes bug 1335172).
- Reorganises the structure of some of the existing non-infrastructure
  docs, to make the documentation easier to navigate.
2019-03-11 21:39:34 +00:00
Ed Morley cd826ef6d9 Bug 1530206 - Remove the bin/run_* scripts
Since they are out of sync with `Procfile` and long term are best replaced
with one of the solutions in bug 1318020.

The development setup instructions have been updated to stop recommending
gunicorn, since Django's runserver is more appropriate for beginners
(and I'm pretty sure the memory leaks seen previously no longer occur).
2019-02-27 07:12:33 +00:00
Ed Morley a0d1f5fccf
Bug 1330474 - Travis: Stop testing against Python 2 (#4675)
Since we've been using Python 3 in production for long enough that we're
confident we won't need to roll back to Python 2.
2019-02-25 06:54:46 +00:00
Ed Morley 86b7b32fd1 Bug 1527868 - Vagrant/Travis: Remove Elasticsearch server
To ensure the CI and local environments match stage/production (where
Elasticsearch is not configured) and don't run differing code-paths due
to the various `ELASTICSEARCH_URL` conditionals used in auto-classify.

At such time when work on the fuzzy auto-classify matcher continues,
this can be reverted.
2019-02-14 21:10:21 +00:00
Cameron Dawson deb71d65e0 Bug 1364045 - Convert all unit tests to Jest (#4603)
Also remove all Karma support and update the docs to only mention ``Jest``.

One of the test files was testing some AngularJS filters. I converted these
tests to test the equivalent helper functions.
2019-02-12 07:49:50 +00:00
Ed Morley fbe255ea86
Vagrant/Travis: Update Elasticsearch to 6.6.0 (#4614)
https://www.elastic.co/guide/en/elasticsearch/reference/6.6/release-notes-6.3.0.html
https://www.elastic.co/guide/en/elasticsearch/reference/6.6/release-notes-6.3.1.html
https://www.elastic.co/guide/en/elasticsearch/reference/6.6/release-notes-6.3.2.html
https://www.elastic.co/guide/en/elasticsearch/reference/6.6/release-notes-6.4.0.html
https://www.elastic.co/guide/en/elasticsearch/reference/6.6/release-notes-6.4.1.html
https://www.elastic.co/guide/en/elasticsearch/reference/6.6/release-notes-6.4.2.html
https://www.elastic.co/guide/en/elasticsearch/reference/6.6/release-notes-6.4.3.html
https://www.elastic.co/guide/en/elasticsearch/reference/6.6/release-notes-6.5.0.html
https://www.elastic.co/guide/en/elasticsearch/reference/6.6/release-notes-6.5.1.html
https://www.elastic.co/guide/en/elasticsearch/reference/6.6/release-notes-6.5.2.html
https://www.elastic.co/guide/en/elasticsearch/reference/6.6/release-notes-6.5.3.html
https://www.elastic.co/guide/en/elasticsearch/reference/6.6/release-notes-6.5.4.html
https://www.elastic.co/guide/en/elasticsearch/reference/6.6/release-notes-6.6.0.html

Note: An Elasticsearch instance does not exist for stage/prod, so this
update is mostly a no-op ready for when/if development on the auto-classify
feature continues.
2019-02-11 11:56:11 +00:00
Ed Morley 979c22710f
Update geckodriver to 0.24.0 (#4610)
https://github.com/mozilla/geckodriver/releases/tag/v0.24.0
2019-02-11 10:21:17 +00:00
Sarah Clements 24bda1ed83
Bug 1508228 - Remove Intermittents Commenter celery task (#4300)
Remove celery task and change Commenter weekly_mode default arg in
preparation for the move to the heroku scheduler
2018-11-27 14:14:38 -08:00
Ed Morley 3decefed0b
Clean up the location of generated output directories (#4289)
So that their purposes are clearer and it's easier to differentiate
between generated content and files committed to the repository.

* Neutrino build: `build/` -> `.build/`
* Sphinx build: `_build/` -> `.build-docs/`
* Django collectstatic: `treeherder/static/` -> `.django-static/`
2018-11-16 19:18:18 +00:00
Ed Morley bb86b7fb6b
Bug 1504617 - Travis: Explicitly start redis-server (#4227)
Since the service is no longer started by default in the base Travis
image as of travis-ci/travis-cookbooks#999.

Fixes:

```
ConnectionError: Error 111 connecting to localhost:6379. Connection refused.
```
(eg https://travis-ci.org/mozilla/treeherder/jobs/450605269#L880)
2018-11-05 09:38:28 +00:00
Ed Morley 565ae4c13e
Bug 1364894 - Upgrade from Neutrino 4 to 9 (#4216)
Neutrino controls our frontend linting, transpilation, source-maps,
testing, dev-server and optimisation of production builds.

Highlights of the upgrade are:

* Major version updates to the individual tools within (such as webpack,
  Babel and ESLint), significantly improving performance, fixing
  transpilation/minification correctness bugs, adding support for newer
  ECMAScript features, and increasing linter coverage.
* Hot reloading in the dev server now works for all entry-points and not
  just the jobs view, shortening the feedback cycle.
* Reduced bundle size due to webpack 4's tree shaking, scope hoisting,
  automatic shared/vendor code chunk splitting (no need for the manually
  maintained 'vendor' list).
* CSS is now extracted out of JS, which improves performance, reduces
  bundle size and prevents the initial white flash of un-styled content.
* Support for dynamic imports/code splitting (needed for bug 1502192).
* Support for Jest via a new Jest preset (unblocks bug 1364045).
* Support for public class field declarations (unblocks bug 1480166).
* Improved source-maps (increases the quality of production exception
  trace-backs and fixes several debugger breakpoint bugs).
* Reduced amount of custom configuration required for our fairly complex
  frontend needs, reducing maintenance burden and allowing for easier
  future Neutrino upgrades.

In addition this PR:

* Fixes the WhiteNoise `immutable_file_test()` regex, so that it now
  correctly enables browser caching of images, fonts and source maps.
* Enables webpack-dev-server's overlay feature, which displays any
  compilation errors in the browser, saving having to switch back
  to the console (this can be enabled for warnings too if desired).
* Enables webpack-dev-server's automatic browser-opening feature,
  which saves having to manually navigate to `localhost:5000` after
  running `yarn start`.
* Switches Karma tests to run Firefox in headless mode, reducing the
  workflow disruption when running `yarn test`.
* Uses the new webpack `performance` option to enable maximum asset
  file size thresholds, to help prevent bundle-size regressions.
* Rewrites the `package.json` script commands so that they now work
  correctly on Windows, even when setting environment variables.

Performance comparison:

* Local `yarn build`:
  - Cached: 2m34s -> 23s
  - Uncached: 2m34s -> 58s
* Local `yarn start`:
  - Cached: 34.5s -> 13.6s
  - Uncached: 34.5s -> 31.3s
* Local `yarn test`
  - Cached: 61.5s -> 19.8s
  - Uncached: 61.5s -> 22.0s
* Local `yarn lint`
  - Cached: 3.8s -> 1.8s
  - Uncached: 13.7s -> 13.4s
* Travis end-to-end time:
  9 minutes -> 6 minutes
* Heroku deploy end-to-end time:
  14 minutes -> 9 minutes
2018-11-02 18:48:28 +00:00
Ed Morley 47419fde6c
Bug 1428031 - Heroku: Increase max load_initial_data retries to 10 (#4203)
Since there was still a deploy failure after five attempts here:
https://dashboard.heroku.com/apps/treeherder-prototype/activity/releases/2847
2018-10-29 09:14:15 +00:00
Ed Morley 165ef15747
Bug 1428031 - Heroku: Retry load_initial_data up to 5 times (#4186)
To work around the intermittent MySQL deadlocks encountered during
a small percentage of deployments. This will ensure deployments don't
cause email spam to app collaborators and Heroku admins, as well as
saving us from having to manually retry the release each time it
occurs.

The `IGNORE_PREDEPLOY_ERRORS` option has also been removed, since
it's mostly redundant with `SKIP_PREDEPLOY`.
2018-10-25 19:07:37 +01:00
Ed Morley 22572cc5ad
Update geckodriver to 0.23.0 (#4100)
https://github.com/mozilla/geckodriver/releases/tag/v0.23.0
2018-10-04 15:19:31 +01:00
Ed Morley 445766d958
Bug 1443251 - Remove support for buildbot job ingestion (#4087)
The buildapi celerybeat tasks were disabled previously in #4007, so
these tasks are unused.
2018-10-02 11:07:27 +01:00
Ed Morley ffaa2e4b2a
Bug 1443251 - Remove runnable jobs support for buildbot (#4071)
Runnable jobs for buildbot were calculated via a celerybeat task
(that was disabled in #4007) and the results stored in the
`runnable_jobs` table. This can all be removed now that buildbot is
EOL, since the remaining support for Taskcluster runnable jobs does
not use that celery task/Django model.
2018-09-27 19:15:47 +01:00
Ed Morley ebc74b9098 Travis: Install a matching version of Yarn
Since Heroku uses the version in `package.json` whereas until now
Travis would use an older version installed in their image.
2018-09-27 00:15:59 +01:00
Ed Morley eae4fa006f
Bug 1492462 - Remove retrigger/cancel APIs and pulse publisher (#4042)
Since as of #3980 (bug 1470622) the frontend no longer calls the
`/retrigger/` `/cancel/` or `/cancel_all/` Treeherder APIs.

Whilst looking at the pulse related fixtures, I spotted that the
`mock_message_broker` fixture was already unused.
2018-09-21 17:39:03 +01:00
Ed Morley 6cd2d3e52e
Update geckodriver to 0.22.0 (#4035)
https://github.com/mozilla/geckodriver/releases/tag/v0.22.0
2018-09-16 21:40:22 +01:00
Ed Morley 817ac227f3
Bug 1489407 - Docs: Pin to older version of CommonMark to fix build (#3998)
CommonMark has made a breaking package change in a minor version
release (0.8.0), so we need to pin to the last working release. See:
https://github.com/rtfd/CommonMark-py/issues/134

Fixes:

```
  ...
  File ".../recommonmark/recommonmark/parser.py", line 9, in <module>
    from CommonMark import Parser
ImportError: No module named CommonMark
```

The Travis docs setup step has also been modified to always upgrade
dependencies, which will mean we will catch cases like this on Travis
next time.
2018-09-07 10:41:04 +01:00
Ed Morley 1b52b07728
Bug 1445325 - Stop mirroring classifications to OrangeFactor (#3677)
Now that consumers of OrangeFactor have been switched to the new
intermittent failures view UI/API, we can stop submitting failure
classifications to OrangeFactor's Elasticsearch instance.
2018-08-03 00:16:35 +02:00
Ed Morley ae0f12f5ba
Bug 1478344 - Travis: Switch from Ubuntu 14.04 to 16.04 (#3826)
* Adds a `rabbitmq-server` install step, since it's no longer
  installed in the image by default.
* Removes the MySQL 5.7 install step, since Ubuntu 16.04's default
  MySQL is now 5.7 rather than 5.6.
* Adjusts which services are manually started to account for changes
  in which are enabled by default in the new image.
2018-07-26 14:26:32 +02:00
Sarah Clements 5386f7ed40
Bug 1453760 - bug commentor (#3571)
* Bug 1453760 - bug commentor
replacement for Orange Factor bug commentor
2018-07-20 16:36:40 -07:00
Ed Morley 5439ddcb6a
Bug 1471182 - Update geckodriver to 0.21.0 (#3717)
https://github.com/mozilla/geckodriver/releases/tag/v0.21.0
2018-06-26 13:05:28 +01:00
Ed Morley 7c48aa1664
Bug 1459148 - Update geckodriver to 0.20.1 (#3520)
https://github.com/mozilla/geckodriver/releases/tag/v0.20.1
2018-05-08 08:22:22 +01:00
George Hickman d5b6cd3e53 Bug 1427598 - Bump Elasticsearch to 6.2.4 2018-04-25 08:37:07 +01:00
Ed Morley 023d31885a
Bug 1453850 - Enable Python 2's `-3` mode on Travis (#3444)
This enables `DeprecationWarnings` for things that Python 2 knows
are not compatible with Python 3. The `error` entry in the pytest
`filterwarnings` setting ensures these will be surfaced as Exceptions
and so result in test failures.

See:
https://docs.python.org/2/using/cmdline.html#cmdoption-3

The removal of `sorted()` from `test_bug_job_map_api.py` is to fix:
`DeprecationWarning: dict inequality comparisons not supported in 3.x`
2018-04-13 23:14:36 +01:00
Ed Morley 5a1b3d60ac
Bug 1445469 - Update to geckodriver v0.20.0 (#3339)
https://github.com/mozilla/geckodriver/releases/tag/v0.20.0
2018-03-13 23:23:33 +00:00
Ed Morley deb1fc86b7 Bug 1384518 - Switch from Memcached to Redis
Since:
* Redis has additional features we need (eg for bug 1409679)
* the Redis server, python client and Django backend are more
  actively maintained (so have persistent connections/pooling that
  actually works, which gives a big perf win)
* we can use Heroku's own Redis addon rather than relying on a
  third-party's (to hopefully prevent a repeat of the certificate
  expiration downtime)

This commit:
* Switches pylibmc/django-pylibmc to redis-py/django-redis, and
  configures the new backend according to:
  http://niwinz.github.io/django-redis/latest/
  https://github.com/andymccurdy/redis-py
* Uses redis-py's native support for TLS to connect to the Heroku
  Redis server's stunnel port directly, avoiding the complexity of
  using a buildpack to create an stunnel between the dyno and server:
  https://devcenter.heroku.com/articles/securing-heroku-redis#connecting-directly-to-stunnel
* Uses explicit `REDIS_URL` values on Travis/Vagrant rather than
  relying on the django-environ `default` value (for parity with
  how we configure `DATABASE_URL` and others).
* Removes the pylibmc connection-closing workaround from `wsgi.py`.

Note: Whilst the Heroku docs suggest using `django-redis-cache`, it's
not as actively maintained, so we're using `django-redis` instead:
https://devcenter.heroku.com/articles/heroku-redis
https://github.com/niwinz/django-redis
https://github.com/sebleier/django-redis-cache

Before this is merged, Heroku Redis instances will need to be created
for stage/production (prototype done) - likely on plan `premium-3`:
https://elements.heroku.com/addons/heroku-redis

We'll also need to pick an eviction policy:
https://devcenter.heroku.com/articles/heroku-redis#maxmemory-policy

Once deployed, the `memcachier-tls-buildpack` buildpack will no longer
be required and can be removed from prototype/stage/prod, along with
the `TREEHERDER_MEMCACHED` environment variable and Memcachier addon.
2018-01-21 14:41:35 +00:00
Ed Morley d9909d5fa2 Bug 1384518 - Add a Redis server instance to Vagrant/Travis
In preparation for later commits, this installs `redis-server` in
the Vagrant environment, and starts the pre-installed (but stopped
by default) `redis-server` on Travis:
https://docs.travis-ci.com/user/database-setup/#Redis

Production/stage/prototype will use Heroku's own Redis add-on:
https://elements.heroku.com/addons/heroku-redis
2018-01-21 14:41:35 +00:00
Ed Morley bc0ca97102
Bug 1419965 - Remove the estimated job time remaining backend (#2990)
The UI has already been removed. This cleans up the data ingestion
and removes the `JobDuration` model, however leaves the `running_eta`
field on the `Job` model for the next time that table is touched (since
the table is large, so altering the schema would likely require
downtime).
2017-12-04 22:09:38 +00:00
Ed Morley e55e58c834 Bug 1420891 - Docs: Test building the docs on Travis 2017-11-28 16:56:45 +00:00
Ed Morley e969648592 Bug 1389572 - Travis: Skip Elasticsearch install if version matches
Since after my update to the Travis images, the pre-installed version
of Elasticsearch matches the one we're looking for, meaning the
download/install can be skipped, saving time.
2017-11-27 16:39:02 +00:00
Ed Morley 4b4a7341c3 Bug 1389572 - Travis: Don't install custom yarn
Since I've adjusted the Travis images to always use the latest yarn
at the time the images are created, so it's not worth installing the
latest version at every runtime:
https://github.com/travis-ci/travis-cookbooks/pull/935
2017-11-27 16:39:02 +00:00
Ed Morley 3f5ef6b352 Bug 1418956 - Set yarn's no-bin-links option via .yarnrc
To save having to pass it each time when using `yarn {install,add}`.
2017-11-20 18:59:26 +00:00
Ed Morley be843a8837 Bug 1414255 - Fix shellcheck warnings about sourced files
```
SC1090: Can't follow non-constant source. Use a directive to specify location.
SC1091: Not following: /etc/profile.d/treeherder.sh was not specified as input (see shellcheck -x).
```

See:
https://github.com/koalaman/shellcheck/wiki/SC1090
https://github.com/koalaman/shellcheck/wiki/SC1091
2017-11-05 20:53:29 +00:00
Ed Morley 37b2c9855e Bug 1414255 - Fix shellcheck warnings about missing quotes
`SC2086: Double quote to prevent globbing and word splitting`

See:
https://github.com/koalaman/shellcheck/wiki/SC2086
2017-11-05 20:53:29 +00:00