зеркало из https://github.com/mozilla/treeherder.git
Bug 1165259 - Add infrastructure documentation (#4766)
- Adds a new "infrastructure" section to the docs, which describes architecture, administration and troubleshooting (fixes bug 1165259). - Adds code comments to any deployment-related files in the repository. - Adds documentation for the various ways in which users can access Treeherder data (fixes bug 1335172). - Reorganises the structure of some of the existing non-infrastructure docs, to make the documentation easier to navigate.
This commit is contained in:
Ed Morley
GitHub
Родитель
783ccb6c9d
Коммит
6945c4c471
|
|
@ -1,3 +1,6 @@
|
|||
// This is the configuration file for Neutrino, which configures webpack and Jest:
|
||||
// https://neutrinojs.org
|
||||
|
||||
// `use strict` is still necessary here since this file is not treated as a module.
|
||||
'use strict'; // eslint-disable-line strict, lines-around-directive
|
||||
|
||||
|
|
|
|||
|
|
@ -1,3 +1,9 @@
|
|||
# Files/directories to be excluded from the Heroku build, to reduce
|
||||
# the resultant slug size and thus speed up dyno deploys. See:
|
||||
# https://devcenter.heroku.com/articles/slug-compiler#ignoring-files-with-slugignore
|
||||
# Anything not committed to version control (for example `node_modules`)
|
||||
# is automatically excluded so does not need to be listed here.
|
||||
|
||||
/docs/
|
||||
/tests/
|
||||
/vagrant/
|
||||
|
|
|
|||
|
|
@ -5,6 +5,7 @@ matrix:
|
|||
include:
|
||||
- env: js-tests
|
||||
language: node_js
|
||||
# The Node version here must be kept in sync with that in `package.json`.
|
||||
node_js: '11.10.1'
|
||||
cache:
|
||||
directories:
|
||||
|
|
@ -20,6 +21,7 @@ matrix:
|
|||
|
||||
- env: python-linters
|
||||
language: python
|
||||
# The Python version here must be kept in sync with that in `runtime.txt` (used by Heroku/Vagrant).
|
||||
python: '3.7.2'
|
||||
cache:
|
||||
directories:
|
||||
|
|
@ -36,6 +38,7 @@ matrix:
|
|||
|
||||
- env: python-tests-main
|
||||
language: python
|
||||
# The Python version here must be kept in sync with that in `runtime.txt` (used by Heroku/Vagrant).
|
||||
python: '3.7.2'
|
||||
cache:
|
||||
directories:
|
||||
|
|
@ -55,12 +58,14 @@ matrix:
|
|||
|
||||
- env: python-tests-selenium
|
||||
language: python
|
||||
# The Python version here must be kept in sync with that in `runtime.txt` (used by Heroku/Vagrant).
|
||||
python: '3.7.2'
|
||||
cache:
|
||||
directories:
|
||||
- ${HOME}/venv
|
||||
- node_modules
|
||||
install:
|
||||
# The Node version here must be kept in sync with that in `package.json`.
|
||||
- nvm install 11
|
||||
- source ./bin/travis-setup.sh services python_env browser js_env
|
||||
before_script:
|
||||
|
|
|
|||
8
Procfile
8
Procfile
|
|
@ -6,7 +6,7 @@
|
|||
# The `release` process type specifies the command to run during deployment, and is where
|
||||
# we run DB migrations and other tasks that are 'release' rather than 'build' specific:
|
||||
# https://devcenter.heroku.com/articles/release-phase
|
||||
# https://12factor.net/build-release-run
|
||||
# https://devcenter.heroku.com/articles/runtime-principles#build-release-run
|
||||
release: ./bin/pre_deploy
|
||||
|
||||
# The `web` process type is the only one that receives external traffic from Heroku's routers.
|
||||
|
|
@ -15,6 +15,7 @@ release: ./bin/pre_deploy
|
|||
# https://devcenter.heroku.com/articles/python-gunicorn
|
||||
# The Heroku Python buildpack sets some sensible gunicorn defaults via environment variables:
|
||||
# https://github.com/heroku/heroku-buildpack-python/blob/master/vendor/python.gunicorn.sh
|
||||
# https://github.com/heroku/heroku-buildpack-python/blob/master/vendor/WEB_CONCURRENCY.sh
|
||||
# TODO: Experiment with different dyno sizes and gunicorn concurrency/worker types (bug 1175472).
|
||||
web: newrelic-admin run-program gunicorn treeherder.config.wsgi:application --timeout 20
|
||||
|
||||
|
|
@ -24,8 +25,9 @@ web: newrelic-admin run-program gunicorn treeherder.config.wsgi:application --ti
|
|||
# The REMAP_SIGTERM is as recommended by:
|
||||
# https://devcenter.heroku.com/articles/celery-heroku#using-remap_sigterm
|
||||
|
||||
# This schedules (but does not run itself) the cron-like tasks listed in `CELERYBEAT_SCHEDULE`.
|
||||
# This schedules (but does not run itself) the cron-like tasks listed in `CELERY_BEAT_SCHEDULE`.
|
||||
# However we're moving away from using this in favour of the Heroku scheduler addon.
|
||||
# NB: This should not be scaled up to more than 1 dyno otherwise duplicate tasks will be scheduled.
|
||||
# TODO: Move the remaining tasks to the addon and remove this process type (deps of bug 1176492).
|
||||
celery_scheduler: REMAP_SIGTERM=SIGQUIT newrelic-admin run-program celery beat -A treeherder
|
||||
|
||||
|
|
@ -33,7 +35,7 @@ celery_scheduler: REMAP_SIGTERM=SIGQUIT newrelic-admin run-program celery beat -
|
|||
# Django management commands. They do not ingest the data themselves, instead adding tasks
|
||||
# to the `store_pulse_{pushes,jobs}` queues for `worker_store_pulse_data` to process.
|
||||
# NB: These should not be scaled up to more than 1 of each.
|
||||
# TODO: Merge these two listeners into one since they use so little CPU each.
|
||||
# TODO: Merge these two listeners into one since they use so little CPU each (bug 1530965).
|
||||
pulse_listener_pushes: newrelic-admin run-program ./manage.py pulse_listener_pushes
|
||||
pulse_listener_jobs: newrelic-admin run-program ./manage.py pulse_listener_jobs
|
||||
|
||||
|
|
|
|||
|
|
@ -1,6 +1,8 @@
|
|||
# -*- mode: ruby -*-
|
||||
# vi: set ft=ruby :
|
||||
|
||||
# TODO: Switch from Vagrant to a docker/docker-compose based environment (bug 1169263).
|
||||
|
||||
# Require a recent Vagrant to reduce the chance of issues being caused by the
|
||||
# use of legacy versions (Vagrant doesn't automatically update on Windows/OS X,
|
||||
# and the ubuntu.com packages are extremely out of date).
|
||||
|
|
|
|||
|
|
@ -1,5 +1,10 @@
|
|||
#!/usr/bin/env bash
|
||||
# Tasks run by the Heroku Python buildpack after the compile step.
|
||||
|
||||
# This script is for running tasks that are 'build' rather than 'release' specific:
|
||||
# https://devcenter.heroku.com/articles/runtime-principles#build-release-run
|
||||
# It is run automatically at the end of the Heroku Python buildpack's compile steps,
|
||||
# which is after pip install and Django collectstatic have been run.
|
||||
# NB: No changes to external services should be made here (use `pre_deploy` instead).
|
||||
|
||||
# Make non-zero exit codes & other errors fatal.
|
||||
set -euo pipefail
|
||||
|
|
@ -10,11 +15,12 @@ echo "$SOURCE_VERSION" > .build/revision.txt
|
|||
# Generate gzipped versions of files that would benefit from compression, that
|
||||
# WhiteNoise can then serve in preference to the originals. This is required
|
||||
# since WhiteNoise's Django storage backend only gzips assets handled by
|
||||
# collectstatic, and so does not affect files in the `.build/` directory.
|
||||
# collectstatic, and so does not affect files in the `.build/` directory
|
||||
# since they are instead generated by Neutrino/webpack.
|
||||
python -m whitenoise.compress .build
|
||||
|
||||
# Remove nodejs files to reduce slug size (and avoid environment variable
|
||||
# pollution from the nodejs profile script), since they are no longer
|
||||
# required after `yarn heroku-postbuild` has run. The buildpack cache will
|
||||
# still contain them, so this doesn't slow down the next slug compile.
|
||||
# Remove nodejs files created by the Heroku Nodejs buildpack, to reduce slug size
|
||||
# (and avoid environment variable pollution from the nodejs profile script), since
|
||||
# they are no longer required after `yarn heroku-postbuild` has run. The buildpack
|
||||
# cache will still contain them, so this doesn't slow down the next slug compile.
|
||||
rm -r .heroku/node/ .heroku/yarn/ .profile.d/nodejs.sh node_modules/
|
||||
|
|
|
|||
|
|
@ -1,6 +1,11 @@
|
|||
#!/usr/bin/env bash
|
||||
# Tasks run after the Heroku buildpack compile, but prior to the deploy.
|
||||
# Failures will block the deploy unless `IGNORE_PREDEPLOY_ERRORS` is set.
|
||||
|
||||
# This script is for running tasks that are 'release' rather than 'build' specific:
|
||||
# https://devcenter.heroku.com/articles/runtime-principles#build-release-run
|
||||
# It is referenced via the `release` entry in `Procfile`, and is run after the
|
||||
# buildpack compile but prior to new code being deployed:
|
||||
# https://devcenter.heroku.com/articles/release-phase
|
||||
# NB: Changes made to the filesystem will not be preserved (use `post_compile` instead).
|
||||
|
||||
# Make non-zero exit codes & other errors fatal.
|
||||
set -euo pipefail
|
||||
|
|
|
|||
|
|
@ -1,5 +1,8 @@
|
|||
#!/usr/bin/env bash
|
||||
|
||||
# Script that is called from `.travis.yml` to bootstrap the Travis environment.
|
||||
# TODO: Use docker in CI instead as part of the move from Vagrant to docker (bug 1169263).
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
# This script must be sourced, so that the environment variables are set in the calling shell.
|
||||
|
|
|
|||
|
|
@ -1,23 +1,8 @@
|
|||
-----BEGIN CERTIFICATE-----
|
||||
MIIDQzCCAqygAwIBAgIJAOd1tlfiGoEoMA0GCSqGSIb3DQEBBQUAMHUxCzAJBgNV
|
||||
BAYTAlVTMRMwEQYDVQQIEwpXYXNoaW5ndG9uMRAwDgYDVQQHEwdTZWF0dGxlMRMw
|
||||
EQYDVQQKEwpBbWF6b24uY29tMQwwCgYDVQQLEwNSRFMxHDAaBgNVBAMTE2F3cy5h
|
||||
bWF6b24uY29tL3Jkcy8wHhcNMTAwNDA1MjI0NDMxWhcNMTUwNDA0MjI0NDMxWjB1
|
||||
MQswCQYDVQQGEwJVUzETMBEGA1UECBMKV2FzaGluZ3RvbjEQMA4GA1UEBxMHU2Vh
|
||||
dHRsZTETMBEGA1UEChMKQW1hem9uLmNvbTEMMAoGA1UECxMDUkRTMRwwGgYDVQQD
|
||||
ExNhd3MuYW1hem9uLmNvbS9yZHMvMIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKB
|
||||
gQDKhXGU7tizxUR5WaFoMTFcxNxa05PEjZaIOEN5ctkWrqYSRov0/nOMoZjqk8bC
|
||||
med9vPFoQGD0OTakPs0jVe3wwmR735hyVwmKIPPsGlaBYj1O6llIpZeQVyupNx56
|
||||
UzqtiLaDzh1KcmfqP3qP2dInzBfJQKjiRudo1FWnpPt33QIDAQABo4HaMIHXMB0G
|
||||
A1UdDgQWBBT/H3x+cqSkR/ePSIinPtc4yWKe3DCBpwYDVR0jBIGfMIGcgBT/H3x+
|
||||
cqSkR/ePSIinPtc4yWKe3KF5pHcwdTELMAkGA1UEBhMCVVMxEzARBgNVBAgTCldh
|
||||
c2hpbmd0b24xEDAOBgNVBAcTB1NlYXR0bGUxEzARBgNVBAoTCkFtYXpvbi5jb20x
|
||||
DDAKBgNVBAsTA1JEUzEcMBoGA1UEAxMTYXdzLmFtYXpvbi5jb20vcmRzL4IJAOd1
|
||||
tlfiGoEoMAwGA1UdEwQFMAMBAf8wDQYJKoZIhvcNAQEFBQADgYEAvguZy/BDT66x
|
||||
GfgnJlyQwnFSeVLQm9u/FIvz4huGjbq9dqnD6h/Gm56QPFdyMEyDiZWaqY6V08lY
|
||||
LTBNb4kcIc9/6pc0/ojKciP5QJRm6OiZ4vgG05nF4fYjhU7WClUx7cxq1fKjNc2J
|
||||
UCmmYqgiVkAGWRETVo+byOSDZ4swb10=
|
||||
-----END CERTIFICATE-----
|
||||
# This is Amazon's RDS public CA certificate, required to make TLS
|
||||
# connections to our RDS instances. See:
|
||||
# https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_MySQL.html#MySQL.Concepts.SSLSupport
|
||||
# https://s3.amazonaws.com/rds-downloads/rds-combined-ca-bundle.pem
|
||||
|
||||
-----BEGIN CERTIFICATE-----
|
||||
MIID9DCCAtygAwIBAgIBQjANBgkqhkiG9w0BAQUFADCBijELMAkGA1UEBhMCVVMx
|
||||
EzARBgNVBAgMCldhc2hpbmd0b24xEDAOBgNVBAcMB1NlYXR0bGUxIjAgBgNVBAoM
|
||||
|
|
@ -258,3 +243,195 @@ NMb0QME981kGRzc2WhgP71YS2hHd1kXtsoYP1yTu4vThSKsoN4bkiHsaC1cRkLoy
|
|||
0fFA4wpB3WloMEvCDaUvvH1LZlBXTNlwi9KtcwD4tDxkkBt4tQczKLGpQ/nF/W9n
|
||||
8YDWk3IIc1sd0bkZqoau2Q==
|
||||
-----END CERTIFICATE-----
|
||||
-----BEGIN CERTIFICATE-----
|
||||
MIIEATCCAumgAwIBAgIBTDANBgkqhkiG9w0BAQUFADCBijELMAkGA1UEBhMCVVMx
|
||||
EzARBgNVBAgMCldhc2hpbmd0b24xEDAOBgNVBAcMB1NlYXR0bGUxIjAgBgNVBAoM
|
||||
GUFtYXpvbiBXZWIgU2VydmljZXMsIEluYy4xEzARBgNVBAsMCkFtYXpvbiBSRFMx
|
||||
GzAZBgNVBAMMEkFtYXpvbiBSRFMgUm9vdCBDQTAeFw0xNTExMDYwMDA1NDZaFw0y
|
||||
MDAzMDUwMDA1NDZaMIGUMQswCQYDVQQGEwJVUzETMBEGA1UECAwKV2FzaGluZ3Rv
|
||||
bjEQMA4GA1UEBwwHU2VhdHRsZTEiMCAGA1UECgwZQW1hem9uIFdlYiBTZXJ2aWNl
|
||||
cywgSW5jLjETMBEGA1UECwwKQW1hem9uIFJEUzElMCMGA1UEAwwcQW1hem9uIFJE
|
||||
UyBhcC1ub3J0aGVhc3QtMiBDQTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoC
|
||||
ggEBAKSwd+RVUzTRH0FgnbwoTK8TMm/zMT4+2BvALpAUe6YXbkisg2goycWuuWLg
|
||||
jOpFBB3GtyvXZnkqi7MkDWUmj1a2kf8l2oLyoaZ+Hm9x/sV+IJzOqPvj1XVUGjP6
|
||||
yYYnPJmUYqvZeI7fEkIGdFkP2m4/sgsSGsFvpD9FK1bL1Kx2UDpYX0kHTtr18Zm/
|
||||
1oN6irqWALSmXMDydb8hE0FB2A1VFyeKE6PnoDj/Y5cPHwPPdEi6/3gkDkSaOG30
|
||||
rWeQfL3pOcKqzbHaWTxMphd0DSL/quZ64Nr+Ly65Q5PRcTrtr55ekOUziuqXwk+o
|
||||
9QpACMwcJ7ROqOznZTqTzSFVXFECAwEAAaNmMGQwDgYDVR0PAQH/BAQDAgEGMBIG
|
||||
A1UdEwEB/wQIMAYBAf8CAQAwHQYDVR0OBBYEFM6Nox/QWbhzWVvzoJ/y0kGpNPK+
|
||||
MB8GA1UdIwQYMBaAFE4C7qw+9hXITO0s9QXBj5yECEmDMA0GCSqGSIb3DQEBBQUA
|
||||
A4IBAQCTkWBqNvyRf3Y/W21DwFx3oT/AIWrHt0BdGZO34tavummXemTH9LZ/mqv9
|
||||
aljt6ZuDtf5DEQjdsAwXMsyo03ffnP7doWm8iaF1+Mui77ot0TmTsP/deyGwukvJ
|
||||
tkxX8bZjDh+EaNauWKr+CYnniNxCQLfFtXYJsfOdVBzK3xNL+Z3ucOQRhr2helWc
|
||||
CDQgwfhP1+3pRVKqHvWCPC4R3fT7RZHuRmZ38kndv476GxRntejh+ePffif78bFI
|
||||
3rIZCPBGobrrUMycafSbyXteoGca/kA+/IqrAPlk0pWQ4aEL0yTWN2h2dnjoD7oX
|
||||
byIuL/g9AGRh97+ssn7D6bDRPTbW
|
||||
-----END CERTIFICATE-----
|
||||
-----BEGIN CERTIFICATE-----
|
||||
MIID/TCCAuWgAwIBAgIBTTANBgkqhkiG9w0BAQsFADCBijELMAkGA1UEBhMCVVMx
|
||||
EzARBgNVBAgMCldhc2hpbmd0b24xEDAOBgNVBAcMB1NlYXR0bGUxIjAgBgNVBAoM
|
||||
GUFtYXpvbiBXZWIgU2VydmljZXMsIEluYy4xEzARBgNVBAsMCkFtYXpvbiBSRFMx
|
||||
GzAZBgNVBAMMEkFtYXpvbiBSRFMgUm9vdCBDQTAeFw0xNjA1MDMyMTI5MjJaFw0y
|
||||
MDAzMDUyMTI5MjJaMIGQMQswCQYDVQQGEwJVUzETMBEGA1UECAwKV2FzaGluZ3Rv
|
||||
bjEQMA4GA1UEBwwHU2VhdHRsZTEiMCAGA1UECgwZQW1hem9uIFdlYiBTZXJ2aWNl
|
||||
cywgSW5jLjETMBEGA1UECwwKQW1hem9uIFJEUzEhMB8GA1UEAwwYQW1hem9uIFJE
|
||||
UyBhcC1zb3V0aC0xIENBMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA
|
||||
06eWGLE0TeqL9kyWOLkS8q0fXO97z+xyBV3DKSB2lg2GkgBz3B98MkmkeB0SZy3G
|
||||
Ce4uCpCPbFKiFEdiUclOlhZsrBuCeaimxLM3Ig2wuenElO/7TqgaYHYUbT3d+VQW
|
||||
GUbLn5GRZJZe1OAClYdOWm7A1CKpuo+cVV1vxbY2nGUQSJPpVn2sT9gnwvjdE60U
|
||||
JGYU/RLCTm8zmZBvlWaNIeKDnreIc4rKn6gUnJ2cQn1ryCVleEeyc3xjYDSrjgdn
|
||||
FLYGcp9mphqVT0byeQMOk0c7RHpxrCSA0V5V6/CreFV2LteK50qcDQzDSM18vWP/
|
||||
p09FoN8O7QrtOeZJzH/lmwIDAQABo2YwZDAOBgNVHQ8BAf8EBAMCAQYwEgYDVR0T
|
||||
AQH/BAgwBgEB/wIBADAdBgNVHQ4EFgQU2i83QHuEl/d0keXF+69HNJph7cMwHwYD
|
||||
VR0jBBgwFoAUTgLurD72FchM7Sz1BcGPnIQISYMwDQYJKoZIhvcNAQELBQADggEB
|
||||
ACqnH2VjApoDqoSQOky52QBwsGaj+xWYHW5Gm7EvCqvQuhWMkeBuD6YJmMvNyA9G
|
||||
I2lh6/o+sUk/RIsbYbxPRdhNPTOgDR9zsNRw6qxaHztq/CEC+mxDCLa3O1hHBaDV
|
||||
BmB3nCZb93BvO0EQSEk7aytKq/f+sjyxqOcs385gintdHGU9uM7gTZHnU9vByJsm
|
||||
/TL07Miq67X0NlhIoo3jAk+xHaeKJdxdKATQp0448P5cY20q4b8aMk1twcNaMvCP
|
||||
dG4M5doaoUA8OQ/0ukLLae/LBxLeTw04q1/a2SyFaVUX2Twbb1S3xVWwLA8vsyGr
|
||||
igXx7B5GgP+IHb6DTjPJAi0=
|
||||
-----END CERTIFICATE-----
|
||||
-----BEGIN CERTIFICATE-----
|
||||
MIID/DCCAuSgAwIBAgIBTjANBgkqhkiG9w0BAQsFADCBijELMAkGA1UEBhMCVVMx
|
||||
EzARBgNVBAgMCldhc2hpbmd0b24xEDAOBgNVBAcMB1NlYXR0bGUxIjAgBgNVBAoM
|
||||
GUFtYXpvbiBXZWIgU2VydmljZXMsIEluYy4xEzARBgNVBAsMCkFtYXpvbiBSRFMx
|
||||
GzAZBgNVBAMMEkFtYXpvbiBSRFMgUm9vdCBDQTAeFw0xNjA4MTExOTU4NDVaFw0y
|
||||
MDAzMDUxOTU4NDVaMIGPMQswCQYDVQQGEwJVUzETMBEGA1UECAwKV2FzaGluZ3Rv
|
||||
bjEQMA4GA1UEBwwHU2VhdHRsZTEiMCAGA1UECgwZQW1hem9uIFdlYiBTZXJ2aWNl
|
||||
cywgSW5jLjETMBEGA1UECwwKQW1hem9uIFJEUzEgMB4GA1UEAwwXQW1hem9uIFJE
|
||||
UyB1cy1lYXN0LTIgQ0EwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCp
|
||||
WnnUX7wM0zzstccX+4iXKJa9GR0a2PpvB1paEX4QRCgfhEdQWDaSqyrWNgdVCKkt
|
||||
1aQkWu5j6VAC2XIG7kKoonm1ZdBVyBLqW5lXNywlaiU9yhJkwo8BR+/OqgE+PLt/
|
||||
EO1mlN0PQudja/XkExCXTO29TG2j7F/O7hox6vTyHNHc0H88zS21uPuBE+jivViS
|
||||
yzj/BkyoQ85hnkues3f9R6gCGdc+J51JbZnmgzUkvXjAEuKhAm9JksVOxcOKUYe5
|
||||
ERhn0U9zjzpfbAITIkul97VVa5IxskFFTHIPJbvRKHJkiF6wTJww/tc9wm+fSCJ1
|
||||
+DbQTGZgkQ3bJrqRN29/AgMBAAGjZjBkMA4GA1UdDwEB/wQEAwIBBjASBgNVHRMB
|
||||
Af8ECDAGAQH/AgEAMB0GA1UdDgQWBBSAHQzUYYZbepwKEMvGdHp8wzHnfDAfBgNV
|
||||
HSMEGDAWgBROAu6sPvYVyEztLPUFwY+chAhJgzANBgkqhkiG9w0BAQsFAAOCAQEA
|
||||
MbaEzSYZ+aZeTBxf8yi0ta8K4RdwEJsEmP6IhFFQHYUtva2Cynl4Q9tZg3RMsybT
|
||||
9mlnSQQlbN/wqIIXbkrcgFcHoXG9Odm/bDtUwwwDaiEhXVfeQom3G77QHOWMTCGK
|
||||
qadwuh5msrb17JdXZoXr4PYHDKP7j0ONfAyFNER2+uecblHfRSpVq5UeF3L6ZJb8
|
||||
fSw/GtAV6an+/0r+Qm+PiI2H5XuZ4GmRJYnGMhqWhBYrY7p3jtVnKcsh39wgfUnW
|
||||
AvZEZG/yhFyAZW0Essa39LiL5VSq14Y1DOj0wgnhSY/9WHxaAo1HB1T9OeZknYbD
|
||||
fl/EGSZ0TEvZkENrXcPlVA==
|
||||
-----END CERTIFICATE-----
|
||||
-----BEGIN CERTIFICATE-----
|
||||
MIID/zCCAuegAwIBAgIBTzANBgkqhkiG9w0BAQsFADCBijELMAkGA1UEBhMCVVMx
|
||||
EzARBgNVBAgMCldhc2hpbmd0b24xEDAOBgNVBAcMB1NlYXR0bGUxIjAgBgNVBAoM
|
||||
GUFtYXpvbiBXZWIgU2VydmljZXMsIEluYy4xEzARBgNVBAsMCkFtYXpvbiBSRFMx
|
||||
GzAZBgNVBAMMEkFtYXpvbiBSRFMgUm9vdCBDQTAeFw0xNjA5MTUwMDEwMTFaFw0y
|
||||
MDAzMDUwMDEwMTFaMIGSMQswCQYDVQQGEwJVUzETMBEGA1UECAwKV2FzaGluZ3Rv
|
||||
bjEQMA4GA1UEBwwHU2VhdHRsZTEiMCAGA1UECgwZQW1hem9uIFdlYiBTZXJ2aWNl
|
||||
cywgSW5jLjETMBEGA1UECwwKQW1hem9uIFJEUzEjMCEGA1UEAwwaQW1hem9uIFJE
|
||||
UyBjYS1jZW50cmFsLTEgQ0EwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIB
|
||||
AQCZYI/iQ6DrS3ny3t1EwX1wAD+3LMgh7Fd01EW5LIuaK2kYIIQpsVKhxLCit/V5
|
||||
AGc/1qiJS1Qz9ODLTh0Na6bZW6EakRzuHJLe32KJtoFYPC7Z09UqzXrpA/XL+1hM
|
||||
P0ZmCWsU7Nn/EmvfBp9zX3dZp6P6ATrvDuYaVFr+SA7aT3FXpBroqBS1fyzUPs+W
|
||||
c6zTR6+yc4zkHX0XQxC5RH6xjgpeRkoOajA/sNo7AQF7KlWmKHbdVF44cvvAhRKZ
|
||||
XaoVs/C4GjkaAEPTCbopYdhzg+KLx9eB2BQnYLRrIOQZtRfbQI2Nbj7p3VsRuOW1
|
||||
tlcks2w1Gb0YC6w6SuIMFkl1AgMBAAGjZjBkMA4GA1UdDwEB/wQEAwIBBjASBgNV
|
||||
HRMBAf8ECDAGAQH/AgEAMB0GA1UdDgQWBBToYWxE1lawl6Ks6NsvpbHQ3GKEtzAf
|
||||
BgNVHSMEGDAWgBROAu6sPvYVyEztLPUFwY+chAhJgzANBgkqhkiG9w0BAQsFAAOC
|
||||
AQEAG/8tQ0ooi3hoQpa5EJz0/E5VYBsAz3YxA2HoIonn0jJyG16bzB4yZt4vNQMA
|
||||
KsNlQ1uwDWYL1nz63axieUUFIxqxl1KmwfhsmLgZ0Hd2mnTPIl2Hw3uj5+wdgGBg
|
||||
agnAZ0bajsBYgD2VGQbqjdk2Qn7Fjy3LEWIvGZx4KyZ99OJ2QxB7JOPdauURAtWA
|
||||
DKYkP4LLJxtj07DSzG8kuRWb9B47uqUD+eKDIyjfjbnzGtd9HqqzYFau7EX3HVD9
|
||||
9Qhnjl7bTZ6YfAEZ3nH2t3Vc0z76XfGh47rd0pNRhMV+xpok75asKf/lNh5mcUrr
|
||||
VKwflyMkQpSbDCmcdJ90N2xEXQ==
|
||||
-----END CERTIFICATE-----
|
||||
-----BEGIN CERTIFICATE-----
|
||||
MIID/DCCAuSgAwIBAgIBUDANBgkqhkiG9w0BAQsFADCBijELMAkGA1UEBhMCVVMx
|
||||
EzARBgNVBAgMCldhc2hpbmd0b24xEDAOBgNVBAcMB1NlYXR0bGUxIjAgBgNVBAoM
|
||||
GUFtYXpvbiBXZWIgU2VydmljZXMsIEluYy4xEzARBgNVBAsMCkFtYXpvbiBSRFMx
|
||||
GzAZBgNVBAMMEkFtYXpvbiBSRFMgUm9vdCBDQTAeFw0xNjEwMTAxNzQ0NDJaFw0y
|
||||
MDAzMDUxNzQ0NDJaMIGPMQswCQYDVQQGEwJVUzETMBEGA1UECAwKV2FzaGluZ3Rv
|
||||
bjEQMA4GA1UEBwwHU2VhdHRsZTEiMCAGA1UECgwZQW1hem9uIFdlYiBTZXJ2aWNl
|
||||
cywgSW5jLjETMBEGA1UECwwKQW1hem9uIFJEUzEgMB4GA1UEAwwXQW1hem9uIFJE
|
||||
UyBldS13ZXN0LTIgQ0EwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDO
|
||||
cttLJfubB4XMMIGWNfJISkIdCMGJyOzLiMJaiWB5GYoXKhEl7YGotpy0qklwW3BQ
|
||||
a0fmVdcCLX+dIuVQ9iFK+ZcK7zwm7HtdDTCHOCKeOh2IcnU4c/VIokFi6Gn8udM6
|
||||
N/Zi5M5OGpVwLVALQU7Yctsn3c95el6MdVx6mJiIPVu7tCVZn88Z2koBQ2gq9P4O
|
||||
Sb249SHFqOb03lYDsaqy1NDsznEOhaRBw7DPJFpvmw1lA3/Y6qrExRI06H2VYR2i
|
||||
7qxwDV50N58fs10n7Ye1IOxTVJsgEA7X6EkRRXqYaM39Z76R894548WHfwXWjUsi
|
||||
MEX0RS0/t1GmnUQjvevDAgMBAAGjZjBkMA4GA1UdDwEB/wQEAwIBBjASBgNVHRMB
|
||||
Af8ECDAGAQH/AgEAMB0GA1UdDgQWBBQBxmcuRSxERYCtNnSr5xNfySokHjAfBgNV
|
||||
HSMEGDAWgBROAu6sPvYVyEztLPUFwY+chAhJgzANBgkqhkiG9w0BAQsFAAOCAQEA
|
||||
UyCUQjsF3nUAABjfEZmpksTuUo07aT3KGYt+EMMFdejnBQ0+2lJJFGtT+CDAk1SD
|
||||
RSgfEBon5vvKEtlnTf9a3pv8WXOAkhfxnryr9FH6NiB8obISHNQNPHn0ljT2/T+I
|
||||
Y6ytfRvKHa0cu3V0NXbJm2B4KEOt4QCDiFxUIX9z6eB4Kditwu05OgQh6KcogOiP
|
||||
JesWxBMXXGoDC1rIYTFO7szwDyOHlCcVXJDNsTJhc32oDWYdeIbW7o/5I+aQsrXZ
|
||||
C96HykZcgWzz6sElrQxUaT3IoMw/5nmw4uWKKnZnxgI9bY4fpQwMeBZ96iHfFxvH
|
||||
mqfEEuC7uUoPofXdBp2ObQ==
|
||||
-----END CERTIFICATE-----
|
||||
-----BEGIN CERTIFICATE-----
|
||||
MIID/DCCAuSgAwIBAgIBUTANBgkqhkiG9w0BAQsFADCBijELMAkGA1UEBhMCVVMx
|
||||
EzARBgNVBAgMCldhc2hpbmd0b24xEDAOBgNVBAcMB1NlYXR0bGUxIjAgBgNVBAoM
|
||||
GUFtYXpvbiBXZWIgU2VydmljZXMsIEluYy4xEzARBgNVBAsMCkFtYXpvbiBSRFMx
|
||||
GzAZBgNVBAMMEkFtYXpvbiBSRFMgUm9vdCBDQTAeFw0xNzA4MjUyMTM5MjZaFw0y
|
||||
MDAzMDUyMTM5MjZaMIGPMQswCQYDVQQGEwJVUzETMBEGA1UECAwKV2FzaGluZ3Rv
|
||||
bjEQMA4GA1UEBwwHU2VhdHRsZTEiMCAGA1UECgwZQW1hem9uIFdlYiBTZXJ2aWNl
|
||||
cywgSW5jLjETMBEGA1UECwwKQW1hem9uIFJEUzEgMB4GA1UEAwwXQW1hem9uIFJE
|
||||
UyBldS13ZXN0LTMgQ0EwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQC+
|
||||
xmlEC/3a4cJH+UPwXCE02lC7Zq5NHd0dn6peMeLN8agb6jW4VfSY0NydjRj2DJZ8
|
||||
K7wV6sub5NUGT1NuFmvSmdbNR2T59KX0p2dVvxmXHHtIpQ9Y8Aq3ZfhmC5q5Bqgw
|
||||
tMA1xayDi7HmoPX3R8kk9ktAZQf6lDeksCvok8idjTu9tiSpDiMwds5BjMsWfyjZ
|
||||
d13PTGGNHYVdP692BSyXzSP1Vj84nJKnciW8tAqwIiadreJt5oXyrCXi8ekUMs80
|
||||
cUTuGm3aA3Q7PB5ljJMPqz0eVddaiIvmTJ9O3Ez3Du/HpImyMzXjkFaf+oNXf/Hx
|
||||
/EW5jCRR6vEiXJcDRDS7AgMBAAGjZjBkMA4GA1UdDwEB/wQEAwIBBjASBgNVHRMB
|
||||
Af8ECDAGAQH/AgEAMB0GA1UdDgQWBBRZ9mRtS5fHk3ZKhG20Oack4cAqMTAfBgNV
|
||||
HSMEGDAWgBROAu6sPvYVyEztLPUFwY+chAhJgzANBgkqhkiG9w0BAQsFAAOCAQEA
|
||||
F/u/9L6ExQwD73F/bhCw7PWcwwqsK1mypIdrjdIsu0JSgwWwGCXmrIspA3n3Dqxq
|
||||
sMhAJD88s9Em7337t+naar2VyLO63MGwjj+vA4mtvQRKq8ScIpiEc7xN6g8HUMsd
|
||||
gPG9lBGfNjuAZsrGJflrko4HyuSM7zHExMjXLH+CXcv/m3lWOZwnIvlVMa4x0Tz0
|
||||
A4fklaawryngzeEjuW6zOiYCzjZtPlP8Fw0SpzppJ8VpQfrZ751RDo4yudmPqoPK
|
||||
5EUe36L8U+oYBXnC5TlYs9bpVv9o5wJQI5qA9oQE2eFWxF1E0AyZ4V5sgGUBStaX
|
||||
BjDDWul0wSo7rt1Tq7XpnA==
|
||||
-----END CERTIFICATE-----
|
||||
-----BEGIN CERTIFICATE-----
|
||||
MIIEATCCAumgAwIBAgIBTjANBgkqhkiG9w0BAQUFADCBijELMAkGA1UEBhMCVVMx
|
||||
EzARBgNVBAgMCldhc2hpbmd0b24xEDAOBgNVBAcMB1NlYXR0bGUxIjAgBgNVBAoM
|
||||
GUFtYXpvbiBXZWIgU2VydmljZXMsIEluYy4xEzARBgNVBAsMCkFtYXpvbiBSRFMx
|
||||
GzAZBgNVBAMMEkFtYXpvbiBSRFMgUm9vdCBDQTAeFw0xNzEyMDEwMDU1NDJaFw0y
|
||||
MDAzMDUwMDU1NDJaMIGUMQswCQYDVQQGEwJVUzETMBEGA1UECAwKV2FzaGluZ3Rv
|
||||
bjEQMA4GA1UEBwwHU2VhdHRsZTEiMCAGA1UECgwZQW1hem9uIFdlYiBTZXJ2aWNl
|
||||
cywgSW5jLjETMBEGA1UECwwKQW1hem9uIFJEUzElMCMGA1UEAwwcQW1hem9uIFJE
|
||||
UyBhcC1ub3J0aGVhc3QtMyBDQTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoC
|
||||
ggEBAMZtQNnm/XT19mTa10ftHLzg5UhajoI65JHv4TQNdGXdsv+CQdGYU49BJ9Eu
|
||||
3bYgiEtTzR2lQe9zGMvtuJobLhOWuavzp7IixoIQcHkFHN6wJ1CvqrxgvJfBq6Hy
|
||||
EuCDCiU+PPDLUNA6XM6Qx3IpHd1wrJkjRB80dhmMSpxmRmx849uFafhN+P1QybsM
|
||||
TI0o48VON2+vj+mNuQTyLMMP8D4odSQHjaoG+zyJfJGZeAyqQyoOUOFEyQaHC3TT
|
||||
3IDSNCQlpxb9LerbCoKu79WFBBq3CS5cYpg8/fsnV2CniRBFFUumBt5z4dhw9RJU
|
||||
qlUXXO1ZyzpGd+c5v6FtrfXtnIUCAwEAAaNmMGQwDgYDVR0PAQH/BAQDAgEGMBIG
|
||||
A1UdEwEB/wQIMAYBAf8CAQAwHQYDVR0OBBYEFETv7ELNplYy/xTeIOInl6nzeiHg
|
||||
MB8GA1UdIwQYMBaAFE4C7qw+9hXITO0s9QXBj5yECEmDMA0GCSqGSIb3DQEBBQUA
|
||||
A4IBAQCpKxOQcd0tEKb3OtsOY8q/MPwTyustGk2Rt7t9G68idADp8IytB7M0SDRo
|
||||
wWZqynEq7orQVKdVOanhEWksNDzGp0+FPAf/KpVvdYCd7ru3+iI+V4ZEp2JFdjuZ
|
||||
Zz0PIjS6AgsZqE5Ri1J+NmfmjGZCPhsHnGZiBaenX6K5VRwwwmLN6xtoqrrfR5zL
|
||||
QfBeeZNJG6KiM3R/DxJ5rAa6Fz+acrhJ60L7HprhB7SFtj1RCijau3+ZwiGmUOMr
|
||||
yKlMv+VgmzSw7o4Hbxy1WVrA6zQsTHHSGf+vkQn2PHvnFMUEu/ZLbTDYFNmTLK91
|
||||
K6o4nMsEvhBKgo4z7H1EqqxXhvN2
|
||||
-----END CERTIFICATE-----
|
||||
-----BEGIN CERTIFICATE-----
|
||||
MIIEBDCCAuygAwIBAgIBTTANBgkqhkiG9w0BAQUFADCBijELMAkGA1UEBhMCVVMx
|
||||
EzARBgNVBAgMCldhc2hpbmd0b24xEDAOBgNVBAcMB1NlYXR0bGUxIjAgBgNVBAoM
|
||||
GUFtYXpvbiBXZWIgU2VydmljZXMsIEluYy4xEzARBgNVBAsMCkFtYXpvbiBSRFMx
|
||||
GzAZBgNVBAMMEkFtYXpvbiBSRFMgUm9vdCBDQTAeFw0xNzEyMDYyMjQyMjdaFw0y
|
||||
MDAzMDQyMjQyMjdaMIGXMQswCQYDVQQGEwJVUzETMBEGA1UECAwKV2FzaGluZ3Rv
|
||||
bjEQMA4GA1UEBwwHU2VhdHRsZTEiMCAGA1UECgwZQW1hem9uIFdlYiBTZXJ2aWNl
|
||||
cywgSW5jLjETMBEGA1UECwwKQW1hem9uIFJEUzEoMCYGA1UEAwwfQW1hem9uIFJE
|
||||
UyBwcmV2aWV3LXVzLWVhc3QtMiBDQTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCC
|
||||
AQoCggEBAMw0E8k8URanS0c/i1S7wzFf5+XC9H2bm+4pENdElGP5s9rVCybrzJaw
|
||||
6zZgVLpOFnS9mJ+sDHIMUexPjj0X4+r7wZ4+hPfy7Rmrgbt23IQwr+PIBxsKAVjj
|
||||
iaQ3bSm5WQ79an5elfQqEDdZ13ckUcLBJDA8bUDthI8m7gnteGtx0M1D0VS5PDs9
|
||||
cf96QlBia9Lx3VcNo3cc0PzP30E4j3h/Ywlb0jXUgB6oVlTxK70BjD3kZa+2xlea
|
||||
vKmm4NqGVhPY7BWd4XNdbSYsPDeZ9HxHNWXZxoHcQ7vSU8RKYVPtoBK/zIp3eWOi
|
||||
gzZlm5vYPvlkYh2pshttPPVyhZqlEZ8CAwEAAaNmMGQwDgYDVR0PAQH/BAQDAgEG
|
||||
MBIGA1UdEwEB/wQIMAYBAf8CAQAwHQYDVR0OBBYEFI93K+FRhste6w3MiD+IK3Tc
|
||||
g/BsMB8GA1UdIwQYMBaAFE4C7qw+9hXITO0s9QXBj5yECEmDMA0GCSqGSIb3DQEB
|
||||
BQUAA4IBAQAs4RsC8MJVOvrlRi5sgKC9LJ4BvSrrbR5V8CdIEwlPqrVOSsU5t7Py
|
||||
j8CHoPUY/ya1azlBSO62BqdZxipFuAR06NdxNG2Gy0fGl71N2udxokwEPW+IEZ81
|
||||
G6JeX8HNFjnna8ehimz1VJDDW7qborhg3dCAgEWkgv5PDR9/zoUu6bbmHPV77zbx
|
||||
Gq7Sybz5OiagC7Nj9N1WgjNXUEmlfY2DHXnJmIVgUGEVrBgu5tGcIU/bQCRznH1N
|
||||
JsBH0SalneCbSzMBhQdnzL+L5KOERibWAZvS6ebmomTBwa03kgo/T0DfEccgobTs
|
||||
rV6T9/8Vg9T18vEeqURL+LOGs7+lIKmN
|
||||
-----END CERTIFICATE-----
|
||||
|
|
@ -0,0 +1,131 @@
|
|||
# Accessing Treeherder's data
|
||||
|
||||
Treeherder's data can be accessed via:
|
||||
|
||||
- [REST API](#rest-api)
|
||||
- [GraphQL API](#graphql-api)
|
||||
- [Redash](#redash)
|
||||
- [ActiveData](#activedata)
|
||||
- [Direct database access](#direct-database-access)
|
||||
|
||||
## REST API
|
||||
|
||||
Treeherder provides a REST API which can be used to query for all the
|
||||
push, job, and performance data it stores internally. For a browsable
|
||||
interface, see:
|
||||
|
||||
<https://treeherder.mozilla.org/docs/>
|
||||
|
||||
### Python Client
|
||||
|
||||
We provide a library, called treeherder-client, to simplify
|
||||
interacting with the REST API. It is maintained inside the
|
||||
Treeherder repository, but you can install your own copy from PyPI
|
||||
using pip:
|
||||
|
||||
```bash
|
||||
pip install treeherder-client
|
||||
```
|
||||
|
||||
It will install a module called `thclient` that you can access, for example:
|
||||
|
||||
```python
|
||||
from thclient import TreeherderClient
|
||||
```
|
||||
|
||||
By default the production Treeherder API will be used, however this can be
|
||||
overridden by passing a `server_url` argument to the `TreeherderClient`
|
||||
constructor:
|
||||
|
||||
```python
|
||||
# Treeherder production
|
||||
client = TreeherderClient()
|
||||
|
||||
# Treeherder stage
|
||||
client = TreeherderClient(server_url='https://treeherder.allizom.org')
|
||||
|
||||
# Local vagrant instance
|
||||
client = TreeherderClient(server_url='http://localhost:8000')
|
||||
```
|
||||
|
||||
The Python client has some convenience methods to query the Treeherder API.
|
||||
|
||||
Here's a simple example which prints the start timestamp of all the
|
||||
jobs associated with the last 10 pushes on mozilla-central:
|
||||
|
||||
```python
|
||||
from thclient import TreeherderClient
|
||||
|
||||
client = TreeherderClient()
|
||||
|
||||
pushes = client.get_pushes('mozilla-central') # gets last 10 by default
|
||||
for pushes in pushes:
|
||||
jobs = client.get_jobs('mozilla-central', push_id=pushes['id'])
|
||||
for job in jobs:
|
||||
print job['start_timestamp']
|
||||
```
|
||||
|
||||
When using the Python client, don't forget to set up logging in the
|
||||
caller so that any API error messages are output, like so:
|
||||
|
||||
```python
|
||||
import logging
|
||||
|
||||
logging.basicConfig()
|
||||
```
|
||||
|
||||
For verbose output, pass `level=logging.DEBUG` to `basicConfig()`.
|
||||
|
||||
### User Agents
|
||||
|
||||
When interacting with Treeherder's API, you must set an appropriate
|
||||
`User Agent` header (rather than relying on the defaults of your
|
||||
language/library) so that we can more easily track API feature usage,
|
||||
as well as accidental abuse. Default scripting User Agents will receive
|
||||
an HTTP 403 response (see [bug 1230222] for more details).
|
||||
|
||||
If you are using the [Python Client](#python-client), an appropriate User Agent
|
||||
is set for you. When using the Python requests library, the User Agent
|
||||
can be set like so:
|
||||
|
||||
```python
|
||||
r = requests.get(url, headers={'User-Agent': ...})
|
||||
```
|
||||
|
||||
[bug 1230222]: https://bugzilla.mozilla.org/show_bug.cgi?id=1230222
|
||||
|
||||
## GraphQL API
|
||||
|
||||
This API is a work in progress. A browsable interface is available at:
|
||||
|
||||
<https://treeherder.mozilla.org/graphql>
|
||||
|
||||
## Redash
|
||||
|
||||
Mozilla's [Redash] instance at <https://sql.telemetry.mozilla.org> is configured to use
|
||||
Treeherder's read-only MySQL RDS replica as a data source. Users with LDAP credentials
|
||||
can find Treeherder's data under the `Treeherder` data source and cross-reference it with
|
||||
other data sets available there.
|
||||
|
||||
[redash]: https://redash.io
|
||||
|
||||
## ActiveData
|
||||
|
||||
[ActiveData] imports Treeherder's production data into its Elasticsearch cluster.
|
||||
See the [getting started with ActiveData] guide for more details.
|
||||
|
||||
[activedata]: https://wiki.mozilla.org/EngineeringProductivity/Projects/ActiveData
|
||||
[getting started with activedata]: https://github.com/mozilla/ActiveData/blob/dev/docs/GettingStarted.md
|
||||
|
||||
## Direct database access
|
||||
|
||||
If there are any use-cases that are not possible via one of the above, we can provide read-only
|
||||
access to Treeherder's production MySQL RDS replica. Please [file an infrastructure bug]
|
||||
requesting that someone from the Treeherder team [grant access to the read-only replica].
|
||||
|
||||
Alternatively if write access is required, we can [create a temporary RDS instance] from
|
||||
a production database snapshot.
|
||||
|
||||
[file an infrastructure bug]: https://bugzilla.mozilla.org/enter_bug.cgi?product=Tree%20Management&component=Treeherder%3A%20Infrastructure
|
||||
[grant access to the read-only replica]: infrastructure/administration.md#granting-access-to-the-read-only-replica
|
||||
[create a temporary rds instance]: infrastructure/administration.md#creating-a-temporary-instance
|
||||
|
|
@ -1,70 +0,0 @@
|
|||
# Administrating Treeherder
|
||||
|
||||
## Direct database access
|
||||
|
||||
For cases where the REST API just isn't enough, a 3rd-party
|
||||
application might want to connect directly to the Treeherder
|
||||
database (or a copy of it). To support these cases, you
|
||||
will probably want to create a specific user for each application
|
||||
who can access publicly available information in a read-only
|
||||
manner (omitting sensitive data like session tokens).
|
||||
|
||||
The following SQL should be sufficient to generate such a user
|
||||
(obviously you should replace `myuser` and `mysecurepassword`):
|
||||
|
||||
```sql
|
||||
CREATE USER 'myuser' IDENTIFIED BY 'mysecurepassword' REQUIRE SSL;
|
||||
|
||||
-- Tables where we want to allow only partial access.
|
||||
-- Whilst `password` is not used (and randomly generated), it's still safer to exclude it.
|
||||
GRANT SELECT (id, username, email) ON treeherder.auth_user to 'myuser';
|
||||
|
||||
-- Tables containing no sensitive data.
|
||||
GRANT SELECT ON treeherder.bug_job_map to 'myuser';
|
||||
GRANT SELECT ON treeherder.bugscache to 'myuser';
|
||||
GRANT SELECT ON treeherder.build_platform to 'myuser';
|
||||
GRANT SELECT ON treeherder.classified_failure to 'myuser';
|
||||
GRANT SELECT ON treeherder.commit to 'myuser';
|
||||
GRANT SELECT ON treeherder.failure_classification to 'myuser';
|
||||
GRANT SELECT ON treeherder.failure_line to 'myuser';
|
||||
GRANT SELECT ON treeherder.group to 'myuser';
|
||||
GRANT SELECT ON treeherder.group_failure_lines to 'myuser';
|
||||
GRANT SELECT ON treeherder.issue_tracker to 'myuser';
|
||||
GRANT SELECT ON treeherder.job to 'myuser';
|
||||
GRANT SELECT ON treeherder.job_detail to 'myuser';
|
||||
GRANT SELECT ON treeherder.job_group to 'myuser';
|
||||
GRANT SELECT ON treeherder.job_log to 'myuser';
|
||||
GRANT SELECT ON treeherder.job_note to 'myuser';
|
||||
GRANT SELECT ON treeherder.job_type to 'myuser';
|
||||
GRANT SELECT ON treeherder.machine to 'myuser';
|
||||
GRANT SELECT ON treeherder.machine_platform to 'myuser';
|
||||
GRANT SELECT ON treeherder.option to 'myuser';
|
||||
GRANT SELECT ON treeherder.option_collection to 'myuser';
|
||||
GRANT SELECT ON treeherder.performance_alert to 'myuser';
|
||||
GRANT SELECT ON treeherder.performance_alert_summary to 'myuser';
|
||||
GRANT SELECT ON treeherder.performance_bug_template to 'myuser';
|
||||
GRANT SELECT ON treeherder.performance_datum to 'myuser';
|
||||
GRANT SELECT ON treeherder.performance_framework to 'myuser';
|
||||
GRANT SELECT ON treeherder.performance_signature to 'myuser';
|
||||
GRANT SELECT ON treeherder.product to 'myuser';
|
||||
GRANT SELECT ON treeherder.push to 'myuser';
|
||||
GRANT SELECT ON treeherder.reference_data_signatures to 'myuser';
|
||||
GRANT SELECT ON treeherder.repository to 'myuser';
|
||||
GRANT SELECT ON treeherder.repository_group to 'myuser';
|
||||
GRANT SELECT ON treeherder.seta_jobpriority to 'myuser';
|
||||
GRANT SELECT ON treeherder.taskcluster_metadata to 'myuser';
|
||||
GRANT SELECT ON treeherder.text_log_error to 'myuser';
|
||||
GRANT SELECT ON treeherder.text_log_error_match to 'myuser';
|
||||
GRANT SELECT ON treeherder.text_log_error_metadata to 'myuser';
|
||||
GRANT SELECT ON treeherder.text_log_step to 'myuser';
|
||||
```
|
||||
|
||||
If new tables are added, you can generate a new set of grant
|
||||
statements using the following SQL:
|
||||
|
||||
```sql
|
||||
SELECT CONCAT('GRANT SELECT ON ', table_schema, '.', table_name, ' to ''myuser'';') AS grant_stmt
|
||||
FROM information_schema.TABLES
|
||||
WHERE table_schema = 'treeherder'
|
||||
AND table_name NOT REGEXP 'django_|auth_';
|
||||
```
|
||||
|
|
@ -66,6 +66,15 @@ To hide jobs we use the job's `tier` setting. Jobs with `tier` of 3 are
|
|||
hidden by default. For TaskCluster, edit the task definition to include the
|
||||
`tier` setting in the Treeherder section.
|
||||
|
||||
## Profiling API endpoint performance
|
||||
|
||||
On our development (vagrant) instance we have [django-debug-toolbar] installed, which can
|
||||
give information on exactly what SQL is run to generate individual API endpoints. Navigate
|
||||
to an endpoint (example: <http://localhost:8000/api/repository/>) and you should see the
|
||||
toolbar to your right.
|
||||
|
||||
[django-debug-toolbar]: https://django-debug-toolbar.readthedocs.io
|
||||
|
||||
## Connecting to Services Running inside Vagrant
|
||||
|
||||
Treeherder uses various services to function, eg MySQL, etc.
|
||||
|
|
|
|||
|
|
@ -1,23 +0,0 @@
|
|||
# Schema Validation
|
||||
|
||||
Some data types in Treeherder will have JSON Schema files in the form of YAML.
|
||||
You can use these files to validate your data prior to submission to be sure
|
||||
it is in the right format.
|
||||
|
||||
You can find all our data schemas in the [schemas] folder.
|
||||
|
||||
To validate your file against a `yml` file, you can use something like the
|
||||
following example code:
|
||||
|
||||
```python
|
||||
import yaml
|
||||
import jsonschema
|
||||
|
||||
schema = yaml.load(open("schemas/text-log-summary-artifact.yml"))
|
||||
jsonschema.validate(data, schema)
|
||||
```
|
||||
|
||||
This will give output telling you if your `data` element passes validation,
|
||||
and, if not, exactly where it is out of compliance.
|
||||
|
||||
[schemas]: https://github.com/mozilla/treeherder/tree/master/schemas
|
||||
|
|
@ -0,0 +1,422 @@
|
|||
# Infrastructure administration
|
||||
|
||||
## Obtaining access
|
||||
|
||||
- Heroku: Follow the [Mozilla Heroku SSO guide] to join the enterprise account,
|
||||
then ask a Treeherder team member who has `manage` permissions to invite you to
|
||||
the Heroku apps.
|
||||
- Amazon RDS: File a [Treeherder infrastructure bug] requesting access to the
|
||||
`moz-devservices` AWS account and needinfo `:dividehex`, who will update the
|
||||
[IAM configuration file][iam-config].
|
||||
- New Relic: The Treeherder team will need to [send an invite][new-relic-invite].
|
||||
- Papertrail: The Treeherder team will need to [send an invite][papertrail-invite].
|
||||
|
||||
[mozilla heroku sso guide]: https://mana.mozilla.org/wiki/display/TS/Using+SSO+with+your+Heroku+account
|
||||
[treeherder infrastructure bug]: https://bugzilla.mozilla.org/enter_bug.cgi?product=Tree%20Management&component=Treeherder%3A%20Infrastructure
|
||||
[new-relic-invite]: https://account.newrelic.com/accounts/677903/users
|
||||
[papertrail-invite]: https://papertrailapp.com/account/members
|
||||
|
||||
## Heroku
|
||||
|
||||
Treeherder has three Heroku apps, which can be managed via the web dashboard or CLI:
|
||||
|
||||
- [treeherder-prototype](https://dashboard.heroku.com/apps/treeherder-prototype)
|
||||
- [treeherder-stage](https://dashboard.heroku.com/apps/treeherder-stage)
|
||||
- [treeherder-prod](https://dashboard.heroku.com/apps/treeherder-prod)
|
||||
|
||||
### Using the Heroku CLI
|
||||
|
||||
Whilst most Heroku administration is possible using the Heroku website, it is often quicker
|
||||
to perform tasks using the [Heroku CLI]. After installing it, run `heroku login` to login
|
||||
(Note: Since Mozilla's Enterprise Heroku account uses SSO, sessions only last 8 hours).
|
||||
|
||||
[heroku cli]: https://devcenter.heroku.com/articles/heroku-cli
|
||||
|
||||
Commands can then be run against a particular app like so:
|
||||
|
||||
```bash
|
||||
$ heroku config --app treeherder-stage
|
||||
```
|
||||
|
||||
For the list of available CLI commands, see the [CLI Usage] page or run `heroku help`.
|
||||
|
||||
[cli usage]: https://devcenter.heroku.com/articles/using-the-cli
|
||||
|
||||
<!-- prettier-ignore -->
|
||||
!!! tip
|
||||
Since Treeherder has multiple Heroku apps, the Heroku CLI feature that allows linking a single
|
||||
app to the local Git repository (to save having to pass `--app` each time) is not helpful.
|
||||
Instead, we recommend adding aliases similar to the following to your bash profile:
|
||||
|
||||
```bash
|
||||
alias thd='HEROKU_APP=treeherder-prototype heroku'
|
||||
alias ths='HEROKU_APP=treeherder-stage heroku'
|
||||
alias thp='HEROKU_APP=treeherder-prod heroku'
|
||||
```
|
||||
|
||||
This allows commands to be run against a specific app with minimal typing:
|
||||
|
||||
```bash
|
||||
$ ths config
|
||||
```
|
||||
|
||||
### Deploying Treeherder
|
||||
|
||||
Deployments occur via Heroku's [GitHub integration] feature, with prototype/stage typically
|
||||
set to auto-deploy from the `master` branch, and production from the `production` branch.
|
||||
This is controlled via the "deploy" tab in the Heroku app dashboard:
|
||||
[prototype][deploy-prototype] | [stage][deploy-stage] | [prod][deploy-prod].
|
||||
A comparison of the Git revisions deployed to each environment can be seen using [What's Deployed].
|
||||
|
||||
After a push is made to an auto-deployed branch, Heroku will wait for the successful completion of
|
||||
the [Travis CI build] (taking approximately 8 minutes), before initiating the deployment process.
|
||||
The steps described in [deployment lifecycle] then occur, which take about 5 minutes.
|
||||
|
||||
Once the deployment is complete, `heroku-bot` will comment in the `#treeherder` IRC channel,
|
||||
and for production, an email is sent to the [tools-treeherder] mailing list. Recent deployment
|
||||
activity can also be seen on the "activity" tab in the Heroku dashboard for each app.
|
||||
|
||||
[github integration]: https://devcenter.heroku.com/articles/github-integration
|
||||
[deploy-prototype]: https://dashboard.heroku.com/apps/treeherder-prototype/deploy/github
|
||||
[deploy-stage]: https://dashboard.heroku.com/apps/treeherder-stage/deploy/github
|
||||
[deploy-prod]: https://dashboard.heroku.com/apps/treeherder-prod/deploy/github
|
||||
[what's deployed]: https://whatsdeployed.io/s-dqv
|
||||
[travis ci build]: https://travis-ci.org/mozilla/treeherder/builds
|
||||
[deployment lifecycle]: architecture.md#deployment-lifecycle
|
||||
[tools-treeherder]: https://lists.mozilla.org/listinfo/tools-treeherder
|
||||
|
||||
<!-- prettier-ignore -->
|
||||
!!! tip
|
||||
To simplify pushing latest `master` to the `production` branch, use this bash alias:
|
||||
|
||||
```bash
|
||||
# Replace `origin` with the remote name of the upstream Treeherder repository, if different.
|
||||
alias deploy='git fetch --all --prune && git push origin remotes/origin/master:production'
|
||||
```
|
||||
|
||||
It pushes directly from the `remotes/origin/master` Git metadata branch, meaning the
|
||||
command works even when the local `master` branch isn't up to date and does not disturb
|
||||
the locally checked out branch or working directory.
|
||||
|
||||
<!-- prettier-ignore -->
|
||||
!!! warning
|
||||
Since we use the GitHub integration feature, never use the `git push heroku master`
|
||||
approach shown in the Heroku tutorials, otherwise the deployed app state won't match
|
||||
the repository branches.
|
||||
|
||||
### Reverting deployments
|
||||
|
||||
Deployments can be reverted by either:
|
||||
|
||||
- Performing a [rollback] using the Heroku web dashboard (via the "activity" tab) or else
|
||||
using the `heroku rollback` CLI command.
|
||||
- Initiating a new deployment with the former code revision.
|
||||
|
||||
Performing a rollback is faster since it re-uses the previously generated app slug so skips
|
||||
the build step. However if auto-deploy from a Git branch (eg `production`) is enabled, then
|
||||
one must remember to fix the issue before the next push to that branch, otherwise the
|
||||
rollback will be overwritten by a newer still-broken release.
|
||||
|
||||
[rollback]: https://devcenter.heroku.com/articles/releases#rollback
|
||||
|
||||
### Restarting dynos
|
||||
|
||||
Heroku's web dashboard can be used to restart all dynos (via the "more" menu top right),
|
||||
or else the [heroku ps:restart] command can be used to restart all/some dyno types.
|
||||
|
||||
[heroku ps:restart]: https://devcenter.heroku.com/articles/heroku-cli-commands#heroku-ps-restart-dyno
|
||||
|
||||
### Scaling dynos
|
||||
|
||||
To change the quantity or [type][dyno-types] (size) of dyno being used for a particular
|
||||
process type, see Heroku's [scaling] documentation.
|
||||
|
||||
If changing the dyno type, it may be necessary to adjust the command's concurrency to make
|
||||
full use of a larger dyno's resources, or conversely to avoid exhausting the RAM of a
|
||||
smaller instance size.
|
||||
|
||||
For gunicorn concurrency is controlled via the `WEB_CONCURRENCY` environment variable, and
|
||||
for Celery via the `--concurrency` CLI option. See the comments in Treeherder's [Procfile]
|
||||
for more details.
|
||||
|
||||
[dyno-types]: https://devcenter.heroku.com/articles/dyno-types
|
||||
[scaling]: https://devcenter.heroku.com/articles/scaling#manual-scaling
|
||||
[procfile]: https://github.com/mozilla/treeherder/blob/master/Procfile
|
||||
|
||||
### Running one-off commands
|
||||
|
||||
Ad-hoc commands can be run against an application using [one-off dynos] that are
|
||||
spun up for the duration of the command and then destroyed after.
|
||||
|
||||
For example to start an interactive bash shell on stage:
|
||||
|
||||
```bash
|
||||
$ heroku run --app treeherder-stage -- bash
|
||||
```
|
||||
|
||||
Or to run a detached Django management command against prod using a larger dyno size:
|
||||
|
||||
```bash
|
||||
$ heroku run:detached --app treeherder-prod --size=standard-2x -- ./manage.py ...
|
||||
```
|
||||
|
||||
[one-off dynos]: https://devcenter.heroku.com/articles/one-off-dynos
|
||||
|
||||
### Resetting the Redis cache
|
||||
|
||||
The Redis cache can be reset by running `./manage.py clear_cache` as a one-off
|
||||
command against the app that owns the [Heroku Redis] add-on in question.
|
||||
|
||||
[heroku redis]: https://devcenter.heroku.com/articles/heroku-redis
|
||||
|
||||
### Adjusting scheduled tasks
|
||||
|
||||
Tasks are run on a schedule via either Heroku's [scheduler addon] or a Celery beat process
|
||||
(see [background tasks] for more details).
|
||||
|
||||
Unfortunately the scheduler addon cannot currently be configured via code or CLI, so changes
|
||||
must be made via the addon's web UI for each Heroku app separately. This can be accessed
|
||||
through the "resources" tab of each Heroku app, or via these direct links:
|
||||
[prototype][scheduler-prototype] | [stage][scheduler-stage] | [prod][scheduler-prod].
|
||||
|
||||
[scheduler addon]: https://devcenter.heroku.com/articles/scheduler
|
||||
[background tasks]: architecture.md#background-tasks
|
||||
[scheduler-prototype]: https://addons-sso.heroku.com/apps/treeherder-prototype/addons/scheduler
|
||||
[scheduler-stage]: https://addons-sso.heroku.com/apps/treeherder-stage/addons/scheduler
|
||||
[scheduler-prod]: https://addons-sso.heroku.com/apps/treeherder-prod/addons/scheduler
|
||||
|
||||
### Environment variables
|
||||
|
||||
App-specific configuration is controlled through the use of environment variables.
|
||||
These can be managed via the "settings" tab of the Heroku app's web dashboard, or with
|
||||
the `heroku config` command. See [managing config variables].
|
||||
|
||||
[managing config variables]: https://devcenter.heroku.com/articles/config-vars#managing-config-vars
|
||||
|
||||
## Amazon RDS
|
||||
|
||||
The MySQL instances used by Treeherder can be found on the [AWS us-east-1 RDS console],
|
||||
after logging in with the account ID `moz-devservices` and then [your IAM username & password].
|
||||
|
||||
[aws us-east-1 rds console]: https://console.aws.amazon.com/rds/home?region=us-east-1#databases:
|
||||
[your iam username & password]: #obtaining-access
|
||||
|
||||
<!-- prettier-ignore -->
|
||||
!!! note
|
||||
For the `treeherder-prod` and `treeherder-stage` Heroku apps, their RDS instances have the
|
||||
same name as the app. However for `treeherder-prototype` the RDS instance is instead called
|
||||
`treeherder-dev`.
|
||||
|
||||
There is also a read-only replica of production, named `treeherder-prod-ro`.
|
||||
|
||||
### Connecting to RDS instances
|
||||
|
||||
Connections **must** be made using TLS otherwise the connection will fail, but not before
|
||||
having already leaked the credentials over plain-text.
|
||||
|
||||
A tool such as [MySQL Workbench] is recommended, since it's possible to save connection
|
||||
settings for each RDS instance, speeding up future use and reducing the chance of forgetting
|
||||
to enable TLS.
|
||||
|
||||
When setting up a connection make sure to change the "Use SSL" option to `require` and set
|
||||
the "SSL CA File" option to point at the AWS public CA certificate, which for convenience can
|
||||
be used [directly from the Treeherder repository][aws-rds-cert]. If using another MySQL client,
|
||||
see the [RDS SSL docs] for more details.
|
||||
|
||||
The public instance hostnames can be found via the [RDS console][aws us-east-1 rds console]
|
||||
or the `DATABASE_URL` environment variable. If accessing production it's strongly recommended
|
||||
to connect to the read-only replica (`treeherder-prod-ro`) to avoid accidental changes or
|
||||
inadvertent DB load, unless write access is specifically required. The replica uses the same
|
||||
credentials as the master production instance.
|
||||
|
||||
[mysql workbench]: https://www.mysql.com/products/workbench/
|
||||
[aws-rds-cert]: https://github.com/mozilla/treeherder/blob/master/deployment/aws/rds-combined-ca-bundle.pem
|
||||
[rds ssl docs]: https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_MySQL.html#MySQL.Concepts.SSLSupport
|
||||
|
||||
### Resetting dev/stage to a recent production snapshot
|
||||
|
||||
The `treeherder-dev` and `treeherder-stage` RDS instances do not have regular user/sheriff
|
||||
interactions that provide certain sets of data (for example classifications of failures).
|
||||
|
||||
As such it can be useful to periodically reset them using the latest daily production snapshot.
|
||||
This has to be performed by an admin of the `moz-devservices` AWS account, and can be requested
|
||||
by filing a bug similar to [bug 1469837].
|
||||
|
||||
After an instance is reset:
|
||||
|
||||
- The password for the MySQL user `th_admin` must be updated to match the one in `DATABASE_URL`
|
||||
for that Heroku app (since the production password will have been inherited from the snapshot,
|
||||
which won't match). See [Changing MySQL passwords].
|
||||
- [Perform a deployment] of the Heroku app, which will ensure that the Django migration and
|
||||
code state are in sync (in case `master` has new migrations not yet seen on production).
|
||||
- [Reset the app's Redis cache]
|
||||
|
||||
[bug 1469837]: https://bugzilla.mozilla.org/show_bug.cgi?id=1469837
|
||||
[changing mysql passwords]: #changing-mysql-passwords
|
||||
[perform a deployment]: #deploying-treeherder
|
||||
[reset the app's redis cache]: #resetting-the-redis-cache
|
||||
|
||||
### Creating a temporary instance
|
||||
|
||||
To create a new RDS instance based on the latest daily production DB snapshot:
|
||||
|
||||
1. Ensure a bug is filed, which will act as a reminder to delete the instance later.
|
||||
2. Go to the [us-east-1 RDS snapshots page].
|
||||
3. Select the most recent `treeherder-prod` snapshot.
|
||||
4. From the "Actions" menu choose "Restore snapshot".
|
||||
5. Set the following options (leaving others at their defaults):
|
||||
- DB instance class: `db.m4.xlarge` (same as dev/stage)
|
||||
- DB instance identifier: `treeherder-dev-bug<NNNNNN>`
|
||||
- Virtual private cloud: `treeherder-prod-vpc`
|
||||
- Subnet group: `treeherder-dbgrp` (press the VPC refresh button to get this to appear)
|
||||
- Public accessibility: Yes
|
||||
- DB parameter group: `treeherder-mysql57`
|
||||
6. Select "Restore DB Instance".
|
||||
7. Wait 15-20 minutes for the instance to be created and report status as "Available".
|
||||
8. Select the instance, click "Modify", and then:
|
||||
- Enter a randomly generated 30+ character password in the "New master password" field.
|
||||
- In the "Security group" section, replace the default group with `treeherder_heroku-sg`.
|
||||
9. Press "Continue" to submit the changes. If the page doesn't submit due to a JavaScript
|
||||
exception, try using the deprecated [classic AWS UI] instead.
|
||||
10. Make a note of the DB hostname recorded under `Endpoint`. It will be in the form
|
||||
`treeherder-dev-bugNNNNNN.HASH.us-east-1.rds.amazonaws.com`.
|
||||
|
||||
To use the new instance locally, run this inside the Vagrant shell:
|
||||
|
||||
```bash
|
||||
export DATABASE_URL='mysql://th_admin:PASSWORD@HOSTNAME/treeherder'
|
||||
```
|
||||
|
||||
Our Django database configuration automatically enables TLS with non-localhost hostnames.
|
||||
|
||||
[us-east-1 rds snapshots page]: https://console.aws.amazon.com/rds/home?region=us-east-1#db-snapshots:
|
||||
[classic aws ui]: https://console.aws.amazon.com/rds/home?region=us-east-1&skin=classic#dbinstances:
|
||||
|
||||
### Granting access to the read-only replica
|
||||
|
||||
One of the ways in which we allow users to [access Treeherder data](../accessing_data.md)
|
||||
is via direct access to our read-only RDS MySQL replica. Both ActiveData and Mozilla's
|
||||
ReDash instance use this approach.
|
||||
|
||||
Each user should be given a unique MySQL username, created by [connecting](#connecting-to-rds-instances)
|
||||
to the master production RDS instance (not the replica) and running these SQL statements:
|
||||
|
||||
```sql
|
||||
-- Adjust the username and password accordingly.
|
||||
CREATE USER 'myuser' IDENTIFIED BY 'PASSWORD >=30 CHARACTERS LONG' REQUIRE SSL;
|
||||
|
||||
-- Tables where we want to allow only partial access.
|
||||
-- `password` is randomly generated by Django and never used/exposed due to SSO,
|
||||
-- so is not really sensitive, but it causes less confusion to still exclude it.
|
||||
GRANT SELECT (id, username, email) ON treeherder.auth_user to 'myuser';
|
||||
|
||||
-- Tables containing no sensitive data.
|
||||
GRANT SELECT ON treeherder.bug_job_map to 'myuser';
|
||||
GRANT SELECT ON treeherder.bugscache to 'myuser';
|
||||
GRANT SELECT ON treeherder.build_platform to 'myuser';
|
||||
GRANT SELECT ON treeherder.classified_failure to 'myuser';
|
||||
GRANT SELECT ON treeherder.commit to 'myuser';
|
||||
GRANT SELECT ON treeherder.failure_classification to 'myuser';
|
||||
GRANT SELECT ON treeherder.failure_line to 'myuser';
|
||||
GRANT SELECT ON treeherder.group to 'myuser';
|
||||
GRANT SELECT ON treeherder.group_failure_lines to 'myuser';
|
||||
GRANT SELECT ON treeherder.issue_tracker to 'myuser';
|
||||
GRANT SELECT ON treeherder.job to 'myuser';
|
||||
GRANT SELECT ON treeherder.job_detail to 'myuser';
|
||||
GRANT SELECT ON treeherder.job_group to 'myuser';
|
||||
GRANT SELECT ON treeherder.job_log to 'myuser';
|
||||
GRANT SELECT ON treeherder.job_note to 'myuser';
|
||||
GRANT SELECT ON treeherder.job_type to 'myuser';
|
||||
GRANT SELECT ON treeherder.machine to 'myuser';
|
||||
GRANT SELECT ON treeherder.machine_platform to 'myuser';
|
||||
GRANT SELECT ON treeherder.option to 'myuser';
|
||||
GRANT SELECT ON treeherder.option_collection to 'myuser';
|
||||
GRANT SELECT ON treeherder.performance_alert to 'myuser';
|
||||
GRANT SELECT ON treeherder.performance_alert_summary to 'myuser';
|
||||
GRANT SELECT ON treeherder.performance_bug_template to 'myuser';
|
||||
GRANT SELECT ON treeherder.performance_datum to 'myuser';
|
||||
GRANT SELECT ON treeherder.performance_framework to 'myuser';
|
||||
GRANT SELECT ON treeherder.performance_signature to 'myuser';
|
||||
GRANT SELECT ON treeherder.product to 'myuser';
|
||||
GRANT SELECT ON treeherder.push to 'myuser';
|
||||
GRANT SELECT ON treeherder.reference_data_signatures to 'myuser';
|
||||
GRANT SELECT ON treeherder.repository to 'myuser';
|
||||
GRANT SELECT ON treeherder.repository_group to 'myuser';
|
||||
GRANT SELECT ON treeherder.seta_jobpriority to 'myuser';
|
||||
GRANT SELECT ON treeherder.taskcluster_metadata to 'myuser';
|
||||
GRANT SELECT ON treeherder.text_log_error to 'myuser';
|
||||
GRANT SELECT ON treeherder.text_log_error_match to 'myuser';
|
||||
GRANT SELECT ON treeherder.text_log_error_metadata to 'myuser';
|
||||
GRANT SELECT ON treeherder.text_log_step to 'myuser';
|
||||
```
|
||||
|
||||
Afterwards provide the user with the newly created credentials and the hostname of the
|
||||
read-only replica (`treeherder-prod-ro.<HASH>.us-east-1.rds.amazonaws.com`), making sure
|
||||
to emphasise the need to [connect using TLS](#connecting-to-rds-instances).
|
||||
|
||||
<!-- prettier-ignore -->
|
||||
!!! warning
|
||||
These credentials will also work on the master production instance, so take care to
|
||||
provide the hostname of the replica and not the master - or their queries will run on
|
||||
the instance used by Treeherder, affecting its performance.
|
||||
|
||||
When tables are added/removed, an updated set of grant statements should be generated using
|
||||
the following SQL:
|
||||
|
||||
```sql
|
||||
SELECT CONCAT('GRANT SELECT ON ', table_schema, '.', table_name, ' to ''myuser'';') AS grant_stmt
|
||||
FROM information_schema.TABLES
|
||||
WHERE table_schema = 'treeherder'
|
||||
AND table_name NOT REGEXP 'django_|auth_';
|
||||
```
|
||||
|
||||
<!-- prettier-ignore -->
|
||||
!!! note
|
||||
For new tables the appropriate `GRANT SELECT` statement will need to be manually run for existing
|
||||
read-only accounts (this is particularly important for the `activedata` and `redash` users).
|
||||
|
||||
### Changing MySQL passwords
|
||||
|
||||
To change the password for a MySQL account on an RDS instance:
|
||||
|
||||
1. [Connect to the instance](#connecting-to-rds-instances) using the `th_admin` user.
|
||||
2. Set the new password [according to the MySQL documentation].
|
||||
|
||||
For example to change the current user's password:
|
||||
|
||||
```sql
|
||||
SET PASSWORD = 'NEW PASSWORD AT LEAST 30 CHARACTERS LONG';
|
||||
```
|
||||
|
||||
Or to change another user's password:
|
||||
|
||||
```sql
|
||||
SET PASSWORD FOR 'another_user' = 'NEW PASSWORD AT LEAST 30 CHARACTERS LONG';
|
||||
```
|
||||
|
||||
3. When changing the `th_admin` password you will need to [update the app's environment variable]
|
||||
`DATABASE_URL` to use the new password. If changing the `activedata` or `redash` user's
|
||||
passwords, the owners of those services will need to be notified.
|
||||
|
||||
<!-- prettier-ignore -->
|
||||
!!! note
|
||||
Whilst the RDS "master account" password can be changed via the AWS console, this can
|
||||
only be performed by an admin of the `moz-devservices` AWS account for stage/prod, so
|
||||
it's easier to change the password using MySQL commands.
|
||||
|
||||
[according to the mysql documentation]: https://dev.mysql.com/doc/refman/5.7/en/set-password.html
|
||||
[update the app's environment variable]: #environment-variables
|
||||
|
||||
### Other changes
|
||||
|
||||
The RDS instances are configured using [Terraform] and [this configuration file][terraform-config],
|
||||
so the IAM permissions [have been set][iam-config] to be very strict (particularly for stage/prod)
|
||||
to prevent the config from drifting out of sync with that in the `devservices-aws` repository.
|
||||
|
||||
To request disk space increases, MySQL configuration changes (via [Parameter Groups]), or MySQL
|
||||
version upgrades, file a [Treeherder infrastructure bug] and needinfo `:dividehex`.
|
||||
|
||||
[terraform]: https://www.terraform.io/
|
||||
[terraform-config]: https://github.com/mozilla-platform-ops/devservices-aws/blob/master/treeherder/rds.tf
|
||||
[iam-config]: https://github.com/mozilla-platform-ops/devservices-aws/blob/master/treeherder/iam.tf
|
||||
[parameter groups]: https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_WorkingWithParamGroups.html
|
||||
|
|
@ -0,0 +1,162 @@
|
|||
# Architecture
|
||||
|
||||
## Overview
|
||||
|
||||
Treeherder is a Django-based Python 3 application that consumes CI data from a variety of
|
||||
sources - making it available via REST and GraphQL APIs to a static Neutrino/webpack-built
|
||||
React frontend.
|
||||
|
||||
Treeherder's [production], [stage] and [prototype] instances are hosted on the [Heroku platform],
|
||||
however [Amazon RDS] is used for their MySQL databases. These RDS instances are owned by the
|
||||
`moz-devservices` AWS account and to reduce latency are in the same AWS region (`us-east-1`) as
|
||||
Heroku's US [common runtime].
|
||||
|
||||
[production]: https://treeherder.mozilla.org
|
||||
[stage]: https://treeherder.allizom.org
|
||||
[prototype]: https://treeherder-prototype.herokuapp.com
|
||||
[heroku platform]: https://www.heroku.com
|
||||
[amazon rds]: https://aws.amazon.com/rds/
|
||||
[common runtime]: https://devcenter.heroku.com/articles/dyno-runtime
|
||||
|
||||
The apps are structured like so:
|
||||
|
||||
<!--
|
||||
Diagram exported from:
|
||||
https://docs.google.com/drawings/d/1Zv8zbdVkpIf8FiDvfc1AaTRhEJed8Pwe7kwAgkTS1zQ/edit
|
||||
-->
|
||||
|
||||
|
||||
|
||||
To learn more about Heroku, see [How Heroku Works] and their [Python app tutorial].
|
||||
|
||||
[how heroku works]: https://devcenter.heroku.com/articles/how-heroku-works
|
||||
[python app tutorial]: https://devcenter.heroku.com/articles/getting-started-with-python
|
||||
|
||||
## Web requests
|
||||
|
||||
Incoming requests are directed to web [dynos] by Heroku's [HTTP router], which also
|
||||
terminates TLS. Certificates are automatically renewed by the Let's Encrypt-powered
|
||||
[Automated Certificate Management] feature.
|
||||
|
||||
On each web dyno, the Django WSGI app is served directly using [gunicorn], which also
|
||||
handles requests for static assets via the Django middleware [WhiteNoise]. In the future
|
||||
the frontend may be hosted separately using Netlify instead, see [bug 1504996].
|
||||
|
||||
[dynos]: https://devcenter.heroku.com/articles/dynos
|
||||
[http router]: https://devcenter.heroku.com/articles/http-routing
|
||||
[automated certificate management]: https://devcenter.heroku.com/articles/automated-certificate-management
|
||||
[gunicorn]: https://gunicorn.org
|
||||
[whitenoise]: http://whitenoise.evans.io
|
||||
[bug 1504996]: https://bugzilla.mozilla.org/show_bug.cgi?id=1504996
|
||||
|
||||
## Background tasks
|
||||
|
||||
Background tasks are primarily managed using [Celery] with [RabbitMQ] as a broker.
|
||||
The RabbitMQ instances are provided by the [CloudAMQP Heroku addon].
|
||||
|
||||
Version control pushes and CI job result data is consumed from Mozilla's event stream, [Pulse],
|
||||
via dynos running [Kombu]-powered Django management commands. These do not ingest the data
|
||||
themselves, instead adding tasks to internal queues for the Celery workers to process.
|
||||
The Celery workers then handle the storing of this push/job data, along with tasks spawned
|
||||
as a result (such as the parsing of logs associated with those jobs).
|
||||
|
||||
For more details on process/worker types, see Treeherder's [Procfile].
|
||||
|
||||
There are also tasks that are run on a schedule, triggered via either:
|
||||
|
||||
1. Heroku's [scheduler addon]
|
||||
|
||||
This runs the commands configured within the addon in [one-off dynos] at the chosen
|
||||
interval (see [adjusting scheduled tasks]).
|
||||
|
||||
The tasks it currently runs are:
|
||||
|
||||
- `update_bugscache` (hourly)
|
||||
- `cycle_data` (daily)
|
||||
- `run_intermittents_commenter` (daily)
|
||||
|
||||
2. The `celery_scheduler` dyno
|
||||
|
||||
This schedules the tasks listed in `CELERY_BEAT_SCHEDULE` in `settings.py`, which are
|
||||
then processed by the `worker_misc` Celery worker dyno. However we're moving away from
|
||||
using this in favour of the Heroku scheduler addon (see [deps of bug 1176492]).
|
||||
|
||||
[celery]: http://celeryproject.org
|
||||
[rabbitmq]: https://www.rabbitmq.com
|
||||
[cloudamqp heroku addon]: https://elements.heroku.com/addons/cloudamqp
|
||||
[pulse]: https://wiki.mozilla.org/Auto-tools/Projects/Pulse
|
||||
[kombu]: https://kombu.readthedocs.io
|
||||
[procfile]: https://github.com/mozilla/treeherder/blob/master/Procfile
|
||||
[scheduler addon]: https://devcenter.heroku.com/articles/scheduler
|
||||
[adjusting scheduled tasks]: administration.md#adjusting-scheduled-tasks
|
||||
[one-off dynos]: https://devcenter.heroku.com/articles/one-off-dynos
|
||||
[deps of bug 1176492]: https://bugzilla.mozilla.org/showdependencytree.cgi?id=1176492&hide_resolved=1
|
||||
|
||||
## Deployment lifecycle
|
||||
|
||||
The Heroku apps are linked to [Treeherder's repository] using Heroku's [GitHub integration],
|
||||
and each instance is normally set to auto-deploy from specific branches (see
|
||||
[Deploying Treeherder]).
|
||||
|
||||
[treeherder's repository]: https://github.com/mozilla/treeherder
|
||||
[github integration]: https://devcenter.heroku.com/articles/github-integration
|
||||
[deploying treeherder]: administration.md#deploying-treeherder
|
||||
|
||||
Once a Treeherder deployment is initiated, the following occurs:
|
||||
|
||||
1. Building the application "slug"
|
||||
|
||||
- The Treeherder source code is checked out from GitHub.
|
||||
- Files matching [.slugignore] entries are removed.
|
||||
- The [buildpacks] configured on the app are invoked, which in our case are:
|
||||
|
||||
1. [heroku-buildpack-nodejs]: Installs Node.js and Yarn (using the versions from
|
||||
`package.json`), runs `yarn install`, then builds the Treeherder frontend via
|
||||
`yarn heroku-postbuild`.
|
||||
|
||||
2. [heroku-buildpack-python]: Installs Python (using the version from `runtime.txt`),
|
||||
pip installs Python dependencies (using `requirements.txt`), for Django apps runs
|
||||
[collectstatic], then triggers our custom [bin/post_compile] script.
|
||||
|
||||
- The results of the above are packaged into a compressed tar archive known as a "slug".
|
||||
|
||||
2. Creating a "release"
|
||||
|
||||
- The slug from the build step is combined with the [config variables] set on the app,
|
||||
to form a [release].
|
||||
- The [Release Phase] feature runs our custom [bin/pre_deploy] script.
|
||||
|
||||
3. Running the new release
|
||||
|
||||
- The existing [dynos] are [shutdown].
|
||||
- New dynos are spun up using the newly-created release with the process types/commands
|
||||
specified in our [Procfile].
|
||||
|
||||
[.slugignore]: https://devcenter.heroku.com/articles/slug-compiler#ignoring-files-with-slugignore
|
||||
[buildpacks]: https://devcenter.heroku.com/articles/buildpacks
|
||||
[heroku-buildpack-nodejs]: https://github.com/heroku/heroku-buildpack-nodejs
|
||||
[heroku-buildpack-python]: https://github.com/heroku/heroku-buildpack-python
|
||||
[collectstatic]: https://docs.djangoproject.com/en/2.1/ref/contrib/staticfiles/#collectstatic
|
||||
[bin/post_compile]: https://github.com/mozilla/treeherder/blob/master/bin/post_compile
|
||||
[config variables]: https://devcenter.heroku.com/articles/config-vars
|
||||
[release]: https://devcenter.heroku.com/articles/releases
|
||||
[release phase]: https://devcenter.heroku.com/articles/release-phase
|
||||
[bin/pre_deploy]: https://github.com/mozilla/treeherder/blob/master/bin/pre_deploy
|
||||
[shutdown]: https://devcenter.heroku.com/articles/dynos#shutdown
|
||||
|
||||
<!-- prettier-ignore -->
|
||||
!!! note
|
||||
Heroku follows the [twelve-factor app] principal of having three distinct phases
|
||||
for the deployment process: [build, release and run]. Whilst all three phases occur
|
||||
for a standard deployment, that is not always the case.
|
||||
|
||||
For example, when [updating environment variables] or [performing a rollback] the build
|
||||
step is skipped and a new release created that re-uses the slug from a previous build.
|
||||
This is why it is important to only interact with external services (such as running
|
||||
DB migrations or notifying New Relic about a deployment) during the release step (using
|
||||
[Release Phase]) and not during the build.
|
||||
|
||||
[twelve-factor app]: https://12factor.net
|
||||
[build, release and run]: https://devcenter.heroku.com/articles/runtime-principles#build-release-run
|
||||
[updating environment variables]: administration.md#environment-variables
|
||||
[performing a rollback]: administration.md#reverting-deployments
|
||||
Различия файлов скрыты, потому что одна или несколько строк слишком длинны
|
После Ширина: | Высота: | Размер: 113 KiB |
|
|
@ -0,0 +1,174 @@
|
|||
# Infrastructure Troubleshooting
|
||||
|
||||
## Monitoring & metrics
|
||||
|
||||
- Heroku apps
|
||||
- Deployment activity:
|
||||
[prototype](https://dashboard.heroku.com/apps/treeherder-prototype/activity) |
|
||||
[stage](https://dashboard.heroku.com/apps/treeherder-stage/activity) |
|
||||
[prod](https://dashboard.heroku.com/apps/treeherder-prod/activity)
|
||||
- HTTP & dyno metrics:
|
||||
[prototype](https://dashboard.heroku.com/apps/treeherder-prototype/metrics) |
|
||||
[stage](https://dashboard.heroku.com/apps/treeherder-stage/metrics) |
|
||||
[prod](https://dashboard.heroku.com/apps/treeherder-prod/metrics)
|
||||
- Service status: <https://status.heroku.com>
|
||||
- New Relic
|
||||
- Overview:
|
||||
[prototype](https://rpm.newrelic.com/accounts/677903/applications/7385291) |
|
||||
[stage](https://rpm.newrelic.com/accounts/677903/applications/14179733) |
|
||||
[prod](https://rpm.newrelic.com/accounts/677903/applications/14179757)
|
||||
- Error analytics:
|
||||
[prototype](https://rpm.newrelic.com/accounts/677903/applications/7385291/filterable_errors) |
|
||||
[stage](https://rpm.newrelic.com/accounts/677903/applications/14179733/filterable_errors) |
|
||||
[prod](https://rpm.newrelic.com/accounts/677903/applications/14179757/filterable_errors)
|
||||
- Web transactions:
|
||||
[prototype](https://rpm.newrelic.com/accounts/677903/applications/7385291/transactions?type=app) |
|
||||
[stage](https://rpm.newrelic.com/accounts/677903/applications/14179733/transactions?type=app) |
|
||||
[prod](https://rpm.newrelic.com/accounts/677903/applications/14179757/transactions?type=app)
|
||||
- Non-web transactions (background tasks):
|
||||
[prototype](https://rpm.newrelic.com/accounts/677903/applications/7385291/transactions?type=other&show_browser=false) |
|
||||
[stage](https://rpm.newrelic.com/accounts/677903/applications/14179733/transactions?type=other&show_browser=false) |
|
||||
[prod](https://rpm.newrelic.com/accounts/677903/applications/14179757/transactions?type=other&show_browser=false)
|
||||
- MySQL/Redis client request stats:
|
||||
[prototype](https://rpm.newrelic.com/accounts/677903/applications/7385291/datastores) |
|
||||
[stage](https://rpm.newrelic.com/accounts/677903/applications/14179733/datastores) |
|
||||
[prod](https://rpm.newrelic.com/accounts/677903/applications/14179757/datastores)
|
||||
- Amazon RDS
|
||||
- CloudWatch metrics:
|
||||
[dev](https://console.aws.amazon.com/rds/home?region=us-east-1#database:id=treeherder-dev;is-cluster=false;tab=monitoring) |
|
||||
[stage](https://console.aws.amazon.com/rds/home?region=us-east-1#database:id=treeherder-stage;is-cluster=false;tab=monitoring) |
|
||||
[prod](https://console.aws.amazon.com/rds/home?region=us-east-1#database:id=treeherder-prod;is-cluster=false;tab=monitoring) |
|
||||
[prod-ro](https://console.aws.amazon.com/rds/home?region=us-east-1#database:id=treeherder-prod-ro;is-cluster=false;tab=monitoring)
|
||||
- Service status: <https://status.aws.amazon.com>
|
||||
- CloudAMQP RabbitMQ add-on
|
||||
- Management & metrics dashboard:
|
||||
[prototype](https://addons-sso.heroku.com/apps/treeherder-prototype/addons/cloudamqp) |
|
||||
[stage](https://addons-sso.heroku.com/apps/treeherder-stage/addons/cloudamqp) |
|
||||
[prod](https://addons-sso.heroku.com/apps/treeherder-prod/addons/cloudamqp)
|
||||
- Service status: <http://status.cloudamqp.com>
|
||||
- Heroku Redis add-on
|
||||
- Management & metrics dashboard:
|
||||
[prototype](https://addons-sso.heroku.com/apps/treeherder-prototype/addons/heroku-redis) |
|
||||
[stage](https://addons-sso.heroku.com/apps/treeherder-stage/addons/heroku-redis) |
|
||||
[prod](https://addons-sso.heroku.com/apps/treeherder-prod/addons/heroku-redis)
|
||||
- Service status: <https://status.heroku.com>
|
||||
|
||||
## Logging
|
||||
|
||||
The Heroku apps are configured to use [Papertrail] as a [log drain]. The aggregated
|
||||
HTTP router, dyno and app output logs can be viewed and searched at:
|
||||
[prototype](https://papertrailapp.com/systems/treeherder-prototype/events) |
|
||||
[stage](https://papertrailapp.com/systems/treeherder-stage/events) |
|
||||
[prod](https://papertrailapp.com/systems/treeherder-prod/events).
|
||||
|
||||
See the Heroku [logging] and [error codes] documentation for help understanding the log output.
|
||||
|
||||
[papertrail]: https://papertrailapp.com
|
||||
[log drain]: https://devcenter.heroku.com/articles/log-drains
|
||||
[logging]: https://devcenter.heroku.com/articles/logging
|
||||
[error codes]: https://devcenter.heroku.com/articles/error-codes
|
||||
|
||||
<!-- prettier-ignore -->
|
||||
!!! note
|
||||
Django is configured to only output log levels `WARNING` and above, unless debug
|
||||
mode is enabled (such in Vagrant).
|
||||
|
||||
## Scenarios
|
||||
|
||||
### Regression from a deployment
|
||||
|
||||
1. Check that the deployment did not involve schema migrations, using the "Compare diff"
|
||||
link on the Heroku app "activity" tab. (If it did, it might be easier to fix the
|
||||
regression in place rather than rolling back the deployment.)
|
||||
2. [Revert the deployment](administration.md#reverting-deployments)
|
||||
3. Notify the rest of the team, so that they do not unknowingly re-deploy the regression.
|
||||
4. If a bug had not already been filed for the regression, either file one now, or re-open
|
||||
the bug that caused the regression, explaining the issue encountered.
|
||||
|
||||
### Web request HTTP 500s
|
||||
|
||||
1. Check the New Relic "Error Analytics" section for details of the Python exception.
|
||||
2. (If needed) Search Papertrail for the exception name/message for additional information.
|
||||
3. File a bug with steps to reproduce and the exception stack trace.
|
||||
|
||||
### Web request HTTP 503s
|
||||
|
||||
If the Heroku [HTTP router] returns an HTTP 503 for a request, it means that a web dyno was
|
||||
unable (or refused) to provide a response. There are several reasons this may occur, which
|
||||
can be differentiated by finding out which [error code][error codes] occurred.
|
||||
|
||||
To discover the error code, check either:
|
||||
|
||||
- the Heroku app "metrics" tab with the "web" dyno type selected, then inspect the timeline.
|
||||
- the Papertrail logs, by searching for `status=503`
|
||||
|
||||
The most common error codes we see, are:
|
||||
|
||||
- [H12 - Request timeout][error-h12]
|
||||
|
||||
This means that the HTTP request took longer than 30 seconds to complete, so was interrupted
|
||||
by the Heroku router. This occurs when a request backlog has formed due to the number or
|
||||
duration of web requests exceeding the capacity of the gunicorn workers available.
|
||||
|
||||
This can be confirmed by looking at the New Relic "Overview" section (making sure the
|
||||
selector at the top of the page is set to "Web transactions time") and checking for spikes
|
||||
in the [Request Queuing] time (which measures how long requests were waiting before a
|
||||
gunicorn worker was able to begin processing them).
|
||||
|
||||
To resolve, first try [restarting] the web dynos in case one of them has an issue (eg: failing
|
||||
dyno or noisy neighbours on that instance). If that doesn't work, check the New Relic web
|
||||
transactions page to see if either there has been a spike in the throughput of requests
|
||||
(in which case consider [scaling] the web dynos), or if an external service (such as MySQL
|
||||
or Redis) is taking longer to respond than before.
|
||||
|
||||
- [H13 - Connection closed without response][error-h13]:
|
||||
|
||||
This means that the request exceeded the gunicorn's configured time limit (currently set
|
||||
at 20 seconds, see the `--timeout` gunicorn argument in `Procfile`), so it aborted the
|
||||
request.
|
||||
|
||||
If most other web requests are succeeding, this suggests that the particular API endpoint
|
||||
and query-parameter combination needs optimisation (in which case file an API bug with
|
||||
steps to reproduce), or if not, try the suggested steps for the `H12` error code above.
|
||||
|
||||
[http router]: https://devcenter.heroku.com/articles/http-routing
|
||||
[error code]: https://devcenter.heroku.com/articles/error-codes
|
||||
[error-h12]: https://devcenter.heroku.com/articles/error-codes#h12-request-timeout
|
||||
[error-h13]: https://devcenter.heroku.com/articles/error-codes#h13-connection-closed-without-response
|
||||
[request queuing]: https://docs.newrelic.com/docs/apm/applications-menu/features/request-queuing-tracking-front-end-time
|
||||
[restarting]: administration.md#restarting-dynos
|
||||
[scaling]: administration.md#scaling-dynos
|
||||
|
||||
### Celery queue backlogs
|
||||
|
||||
When the RabbitMQ queues used by Celery exceeds the configured threshold, CloudAMQP sends
|
||||
an email alert to the `treeherder-internal@moco` mailing list. If this occurs:
|
||||
|
||||
1. Open the CloudAMQP management dashboard using the links above, then:
|
||||
- Check the "Metrics" tab, to see a timeline of total queue length.
|
||||
- Click the "RabbitMQ Manager" button, and switch to the "Queues" tab to see
|
||||
the per-queue breakdown and message incoming/delivery rates.
|
||||
2. Check New Relic's "Error Analytics" section, in case tasks are failing and being
|
||||
retried due to a Python exception.
|
||||
3. In the New Relic's "Transactions" section, switch to the "Non-web" transactions view
|
||||
(or use the direct links above), and click the relevant Celery task to see if there
|
||||
has been a change in either throughput or time per task.
|
||||
4. Depending on the information discovered above, you may want to try [restarting] or
|
||||
[scaling] the worker dynos associated with the backlogged queues.
|
||||
|
||||
### New pushes/jobs not appearing
|
||||
|
||||
If new pushes or CI job results are not appearing in Treeherder's UI:
|
||||
|
||||
1. Follow the steps in [Celery queue backlogs](#celery-queue-backlogs) to rule out
|
||||
task backlogs/Python exceptions.
|
||||
2. Check the upstream Pulse queues [using Pulse Guardian] (you must be an co-owner of
|
||||
the Treeherder queues to see them listed). If there is a Pulse queue backlog,
|
||||
it suggests that Treeherder's `pulse_listener_{pushes,jobs}` dynos have stopped
|
||||
consuming Pulse events, and so might need [restarting].
|
||||
3. Failing that, it's possible the issue might lie in the services that send events to
|
||||
those Pulse exchanges, such as `taskcluster-treeherder`, `taskcluster-github` or
|
||||
the Taskcluster systems upstream of those. Ask for help in the IRC channel
|
||||
`#taskcluster`.
|
||||
|
||||
[using pulse guardian]: https://pulseguardian.mozilla.org/queues
|
||||
|
|
@ -72,7 +72,7 @@ See the [code style](code_style.md#ui) section for more details.
|
|||
|
||||
### Running the unit tests
|
||||
|
||||
The unit tests for the UI are run with [Jest] and [Jasmine]. React components are tested with [enzyme].
|
||||
The unit tests for the UI are run with [Jest].
|
||||
|
||||
To run the tests:
|
||||
|
||||
|
|
@ -105,7 +105,7 @@ To get started:
|
|||
|
||||
It will typically take 5 to 30 minutes for the vagrant provision to
|
||||
complete, depending on your network performance. If you experience
|
||||
any errors, see the [troubleshooting page](troubleshooting.md).
|
||||
any errors, see the [Vagrant troubleshooting page](troubleshooting_vagrant.md).
|
||||
|
||||
It is _very important_ that the provisioning process complete successfully before
|
||||
trying to interact with your test instance of treeherder: some things might
|
||||
|
|
@ -226,5 +226,3 @@ Continue to **Working with the Server** section after looking at the [Code Style
|
|||
[yarn]: https://yarnpkg.com/en/docs/install
|
||||
[package.json]: https://github.com/mozilla/treeherder/blob/master/package.json
|
||||
[eslint]: https://eslint.org
|
||||
[jasmine]: https://jasmine.github.io/
|
||||
[enzyme]: http://airbnb.io/enzyme/
|
||||
|
|
|
|||
|
|
@ -1,75 +0,0 @@
|
|||
# REST API
|
||||
|
||||
Treeherder provides a REST API which can be used to query for all the
|
||||
push, job, and performance data it stores internally. For a browsable
|
||||
interface, see:
|
||||
<https://treeherder.mozilla.org/docs/>
|
||||
|
||||
## Profiling API endpoint performance
|
||||
|
||||
On our development (vagrant) instance we have [django-debug-toolbar](http://django-debug-toolbar.readthedocs.io/) installed, which can give
|
||||
information on exactly what SQL is run to generate individual API
|
||||
endpoints. Just navigate to an endpoint
|
||||
(example: <http://localhost:8000/api/repository/>) and
|
||||
you should see the toolbar to your right.
|
||||
|
||||
## Python Client
|
||||
|
||||
We provide a library, called treeherder-client, to simplify
|
||||
interacting with the REST API. It is maintained inside the
|
||||
Treeherder repository, but you can install your own copy from PyPI
|
||||
using pip:
|
||||
|
||||
```bash
|
||||
pip install treeherder-client
|
||||
```
|
||||
|
||||
It will install a module called `thclient` that you can access, for example:
|
||||
|
||||
```python
|
||||
from thclient import TreeherderClient
|
||||
```
|
||||
|
||||
By default the production Treeherder API will be used, however this can be
|
||||
overridden by passing a `server_url` argument to the `TreeherderClient`
|
||||
constructor:
|
||||
|
||||
```python
|
||||
# Treeherder production
|
||||
client = TreeherderClient()
|
||||
|
||||
# Treeherder stage
|
||||
client = TreeherderClient(server_url='https://treeherder.allizom.org')
|
||||
|
||||
# Local vagrant instance
|
||||
client = TreeherderClient(server_url='http://localhost:8000')
|
||||
```
|
||||
|
||||
When using the Python client, don't forget to set up logging in the
|
||||
caller so that any API error messages are output, like so:
|
||||
|
||||
```python
|
||||
import logging
|
||||
|
||||
logging.basicConfig()
|
||||
```
|
||||
|
||||
For verbose output, pass `level=logging.DEBUG` to `basicConfig()`.
|
||||
|
||||
## User Agents
|
||||
|
||||
When interacting with Treeherder's API, you must set an appropriate
|
||||
`User Agent` header (rather than relying on the defaults of your
|
||||
language/library) so that we can more easily track API feature usage,
|
||||
as well as accidental abuse. Default scripting User Agents will receive
|
||||
an HTTP 403 response (see [bug 1230222] for more details).
|
||||
|
||||
If you are using the [Python Client](#python-client), an appropriate User Agent
|
||||
is set for you. When using the Python requests library, the User Agent
|
||||
can be set like so:
|
||||
|
||||
```python
|
||||
r = requests.get(url, headers={'User-Agent': ...})
|
||||
```
|
||||
|
||||
[bug 1230222]: https://bugzilla.mozilla.org/show_bug.cgi?id=1230222
|
||||
|
|
@ -1,19 +0,0 @@
|
|||
# Retrieving Data
|
||||
|
||||
The [Python client](rest_api.md#python-client) also has some convenience
|
||||
methods to query the Treeherder API.
|
||||
|
||||
Here's a simple example which prints the start timestamp of all the
|
||||
jobs associated with the last 10 pushes on mozilla-central:
|
||||
|
||||
```python
|
||||
from thclient import TreeherderClient
|
||||
|
||||
client = TreeherderClient()
|
||||
|
||||
pushes = client.get_pushes('mozilla-central') # gets last 10 by default
|
||||
for pushes in pushes:
|
||||
jobs = client.get_jobs('mozilla-central', push_id=pushes['id'])
|
||||
for job in jobs:
|
||||
print job['start_timestamp']
|
||||
```
|
||||
|
|
@ -1,4 +1,4 @@
|
|||
# Submitting Data
|
||||
# Submitting data to Treeherder
|
||||
|
||||
To submit your test data to Treeherder, you have two options:
|
||||
|
||||
|
|
@ -106,6 +106,32 @@ to your Exchange and they will start showing in Treeherder.
|
|||
You will no longer need any special credentials. You publish messages to the
|
||||
Exchange YOU own. Treeherder is now just listening to it.
|
||||
|
||||
## Schema Validation
|
||||
|
||||
Some data types in Treeherder will have JSON Schema files in the form of YAML.
|
||||
You can use these files to validate your data prior to submission to be sure
|
||||
it is in the right format.
|
||||
|
||||
You can find all our data schemas in the [schemas] folder.
|
||||
|
||||
To validate your file against a `yml` file, you can use something like the
|
||||
following example code:
|
||||
|
||||
```python
|
||||
import yaml
|
||||
import jsonschema
|
||||
|
||||
with open('schemas/text-log-summary-artifact.yml') as f:
|
||||
schema = yaml.load(f)
|
||||
|
||||
jsonschema.validate(data, schema)
|
||||
```
|
||||
|
||||
This will give output telling you if your `data` element passes validation,
|
||||
and, if not, exactly where it is out of compliance.
|
||||
|
||||
[schemas]: https://github.com/mozilla/treeherder/tree/master/schemas
|
||||
|
||||
## Adding a GitHub Repository
|
||||
|
||||
The pushes from GitHub repos come to Treeherder via Pulse. The webhook to enable
|
||||
|
|
|
|||
|
|
@ -1,4 +1,4 @@
|
|||
# Troubleshooting
|
||||
# Troubleshooting Vagrant
|
||||
|
||||
## Errors during Vagrant setup
|
||||
|
||||
22
mkdocs.yml
22
mkdocs.yml
|
|
@ -32,18 +32,18 @@ markdown_extensions:
|
|||
# http://www.mkdocs.org/user-guide/writing-your-docs/#configure-pages-and-navigation
|
||||
nav:
|
||||
- About: 'index.md'
|
||||
- Getting Started:
|
||||
- Development:
|
||||
- Installation: 'installation.md'
|
||||
- Loading Pulse data: 'pulseload.md'
|
||||
- Code Style: 'code_style.md'
|
||||
- Common tasks: 'common_tasks.md'
|
||||
- Backend tasks: 'backend_tasks.md'
|
||||
- Working with the Server:
|
||||
- Loading Pulse data: 'pulseload.md'
|
||||
- SETA: 'seta.md'
|
||||
- REST API: 'rest_api.md'
|
||||
- Submitting Data: 'submitting_data.md'
|
||||
- Retrieving Data: 'retrieving_data.md'
|
||||
- Schema Validation: 'data_validation.md'
|
||||
- Troubleshooting: 'troubleshooting.md'
|
||||
- Treeherder Test Cases: 'testcases.md'
|
||||
- Administrating Treeherder: 'admin.md'
|
||||
- Troubleshooting Vagrant: 'troubleshooting_vagrant.md'
|
||||
- Infrastructure:
|
||||
- Architecture: 'infrastructure/architecture.md'
|
||||
- Administration: 'infrastructure/administration.md'
|
||||
- Troubleshooting: 'infrastructure/troubleshooting.md'
|
||||
- Accessing data: 'accessing_data.md'
|
||||
- Submitting data: 'submitting_data.md'
|
||||
- SETA: 'seta.md'
|
||||
- Manual test cases: 'testcases.md'
|
||||
|
|
|
|||
|
|
@ -1,3 +1,5 @@
|
|||
# HEROKU REQUIREMENTS
|
||||
# Our pip requirements files are kept under `requirements/`, however Heroku's
|
||||
# Python buildpack only looks for `<repo_root>/requirements.txt`, so we use
|
||||
# pip's include syntax to load the production dependencies file from here.
|
||||
|
||||
-r requirements/common.txt
|
||||
|
|
|
|||
|
|
@ -111,8 +111,10 @@ for alias in DATABASES:
|
|||
}
|
||||
if DATABASES[alias]['HOST'] != 'localhost':
|
||||
# Use TLS when connecting to RDS.
|
||||
# https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_MySQL.html#MySQL.Concepts.SSLSupport
|
||||
# https://mysqlclient.readthedocs.io/user_guide.html#functions-and-attributes
|
||||
DATABASES[alias]['OPTIONS']['ssl'] = {
|
||||
'ca': 'deployment/aws/combined-ca-bundle.pem',
|
||||
'ca': 'deployment/aws/rds-combined-ca-bundle.pem',
|
||||
}
|
||||
|
||||
# Caches
|
||||
|
|
@ -143,6 +145,7 @@ STATIC_URL = "/static/"
|
|||
|
||||
# Create hashed+gzipped versions of assets during collectstatic,
|
||||
# which will then be served by WhiteNoise with a suitable max-age.
|
||||
# http://whitenoise.evans.io/en/stable/django.html#add-compression-and-caching-support
|
||||
STATICFILES_STORAGE = 'whitenoise.storage.CompressedManifestStaticFilesStorage'
|
||||
|
||||
# Authentication
|
||||
|
|
@ -372,6 +375,7 @@ REST_FRAMEWORK = {
|
|||
}
|
||||
|
||||
# Whitenoise
|
||||
# http://whitenoise.evans.io/en/stable/django.html#available-settings
|
||||
# Files in this directory will be served by WhiteNoise at the site root.
|
||||
WHITENOISE_ROOT = join(SRC_DIR, ".build")
|
||||
# Serve index.html for URLs ending in a trailing slash.
|
||||
|
|
|
|||
|
|
@ -10,10 +10,6 @@ import os
|
|||
|
||||
from django.core.wsgi import get_wsgi_application
|
||||
|
||||
# We defer to a DJANGO_SETTINGS_MODULE already in the environment. This breaks
|
||||
# if running multiple sites in the same mod_wsgi process. To fix this, use
|
||||
# mod_wsgi daemon mode with each site in its own daemon process, or use
|
||||
# os.environ["DJANGO_SETTINGS_MODULE"] = "treeherder.config.settings"
|
||||
os.environ.setdefault("DJANGO_SETTINGS_MODULE", "treeherder.config.settings")
|
||||
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'treeherder.config.settings')
|
||||
|
||||
application = get_wsgi_application()
|
||||
|
|
|
|||
|
|
@ -28,7 +28,15 @@ CSP_HEADER = '; '.join(CSP_DIRECTIVES)
|
|||
|
||||
|
||||
class CustomWhiteNoise(WhiteNoiseMiddleware):
|
||||
"""Sets long max-age headers for Neutrino-generated hashed files."""
|
||||
"""
|
||||
Extends WhiteNoiseMiddleware with two additional features:
|
||||
1) Adds a `Content-Security-Policy` header to all static file responses.
|
||||
2) Allows WhiteNoise to recognise Neutrino-generated hashed filenames as "immutable",
|
||||
so that WhiteNoise will then set long Cache-Control max-age headers for them.
|
||||
|
||||
For the stock functionality provided by WhiteNoiseMiddleware see:
|
||||
http://whitenoise.evans.io/
|
||||
"""
|
||||
|
||||
# Matches Neutrino's style of hashed filename URLs, eg:
|
||||
# /assets/index.1d85033a.js
|
||||
|
|
|
|||
|
|
@ -52,8 +52,7 @@ const menuItems = [
|
|||
text: 'Source',
|
||||
},
|
||||
{
|
||||
href:
|
||||
'https://whatsdeployed.io/?owner=mozilla&repo=treeherder&name[]=Stage&url[]=https://treeherder.allizom.org/revision.txt&name[]=Prod&url[]=https://treeherder.mozilla.org/revision.txt',
|
||||
href: 'https://whatsdeployed.io/s-dqv',
|
||||
icon: faQuestion,
|
||||
text: "What's Deployed?",
|
||||
},
|
||||
|
|
|
|||
|
|
@ -27,10 +27,7 @@ const UserGuideFooter = function UserGuideFooter() {
|
|||
</div>
|
||||
|
||||
<div>
|
||||
<a
|
||||
className="midgray"
|
||||
href="http://whatsdeployed.io/?owner=mozilla&repo=treeherder&name[]=Stage&url[]=https://treeherder.allizom.org/revision.txt&name[]=Prod&url[]=https://treeherder.mozilla.org/revision.txt"
|
||||
>
|
||||
<a className="midgray" href="https://whatsdeployed.io/s-dqv">
|
||||
Whats Deployed?
|
||||
</a>
|
||||
</div>
|
||||
|
|
|
|||
|
|
@ -1,5 +1,7 @@
|
|||
#!/usr/bin/env bash
|
||||
|
||||
# Script that is run during Vagrant provision to set up the development environment.
|
||||
# TODO: Switch from Vagrant to a docker/docker-compose based environment (bug 1169263).
|
||||
|
||||
# Make non-zero exit codes & other errors fatal.
|
||||
set -euo pipefail
|
||||
|
|
|
|||
Загрузка…
Ссылка в новой задаче