Add markdownlint pre-commit hook and fixed docs

We also add it as part of the Travis execution.
This commit is contained in:
Shubham Kumar 2020-04-04 07:29:36 +05:30 коммит произвёл Armen Zambrano G
Родитель c7f9fbb5de
Коммит 10a2040774
14 изменённых файлов: 108 добавлений и 52 удалений

4
.markdownlint.json Normal file
Просмотреть файл

@ -0,0 +1,4 @@
{
"default": true,
"MD013": false
}

1
.markdownlintignore Normal file
Просмотреть файл

@ -0,0 +1 @@
docs/index.md

Просмотреть файл

@ -20,3 +20,7 @@ repos:
rev: 1.19.1
hooks:
- id: prettier
- repo: https://github.com/igorshubovych/markdownlint-cli
rev: v0.22.0
hooks:
- id: markdownlint

Просмотреть файл

@ -66,6 +66,7 @@ jobs:
- pip install -r requirements/dev.txt
- pip install -r requirements/common.txt
script:
- pre-commit run --all-files markdownlint
- ./runchecks.sh
- ./manage.py check
# Several security features in settings.py (eg setting HSTS headers) are conditional on

Просмотреть файл

@ -6,21 +6,21 @@
[![Node devDependencies Status](https://david-dm.org/mozilla/treeherder/dev-status.svg)](https://david-dm.org/mozilla/treeherder?type=dev)
[![Documentation Status](https://readthedocs.org/projects/treeherder/badge/?version=latest)](https://treeherder.readthedocs.io/?badge=latest)
#### Description
## Description
[Treeherder](https://treeherder.mozilla.org) is a reporting dashboard for Mozilla checkins. It allows users to see the results of automatic builds and their respective tests. The Treeherder service manages the etl layer for data ingestion, web services, and the data model behind Treeherder.
#### Instances
## Instances
Treeherder exists on two instances: [staging](https://treeherder.allizom.org) for pre-deployment validation, and [production](https://treeherder.mozilla.org) for actual use.
#### Installation
## Installation
The steps to run Treeherder are provided [here](https://treeherder.readthedocs.io/installation.html).
The steps to run only the UI are provided [here](https://treeherder.readthedocs.io/installation.html#ui-development).
#### Links
## Links
Visit our project tracking Wiki [here](https://wiki.mozilla.org/EngineeringProductivity/Projects/Treeherder).
@ -28,8 +28,7 @@ For other setup and configuration, visit our readthedocs page [here](https://t
File any bugs you may encounter [here](https://bugzilla.mozilla.org/enter_bug.cgi?product=Tree+Management&component=Treeherder).
#### Contributing
## Contributing
Everyone is welcome to contribute!
@ -41,7 +40,8 @@ After adressing the issue, make sure [every test passes](https://treeherder.read
We also recommend setting an `upstream` remote that points to the [Mozilla's Github repo](https://github.com/mozilla/treeherder.git), in addition to `origin` that points to your fork. You should then frequently use `git rebase upstream` rather than merging from your fork to keep your branch current. There are less conflicts this way and the git history is cleaner.
##### Sending a Pull Request
## Sending a Pull Request
We receive contributions from both Bugzilla and Github. We have some specifications to keep track of them:
1. If your bug comes from **[Bugzilla](https://bugzilla.mozilla.org/query.cgi?query_format=advanced&product=Tree+Management&f1=component&o1=substring&v1=Treeherder&resolution=---)**
@ -55,7 +55,7 @@ We receive contributions from both Bugzilla and Github. We have some specificati
2. If your bug comes from **Github**
In the **description** of the pull request, please mention the **issue number**. That can be done by typing #[issue's number].
For example: "This pull request fixes #5135".
Github automatically links both issue and pull request to one another.

Просмотреть файл

@ -120,7 +120,7 @@ See the [getting started with ActiveData] guide for more details.
## Direct database access
If the use-cases above aren't sufficient or you're working on a fullstack Perfherder bug,
If the use-cases above aren't sufficient or you're working on a fullstack Perfherder bug,
we can provide read-only access to Treeherder's production MySQL RDS replica.
Please [file an infrastructure bug] requesting that someone from the
Treeherder team [grant access to the read-only replica].

Просмотреть файл

@ -18,6 +18,7 @@ Or if you would rather not use Docker, instead use poetry,run:
% poetry install
% poetry run mkdocs serve
```
**Note** - On Windows you might need to fallback ```python -m venv venv``` or ```virtualenv``` to manage your virtualenv if ```poetry``` does not work for you.
In either case, the docs will then be available at: <http://localhost:8000>

Просмотреть файл

@ -35,7 +35,7 @@ to perform tasks using the [Heroku CLI]. After installing it, run `heroku login`
Commands can then be run against a particular app like so:
```bash
$ heroku config --app treeherder-stage
heroku config --app treeherder-stage
```
For the list of available CLI commands, see the [CLI Usage] page or run `heroku help`.
@ -48,17 +48,17 @@ For the list of available CLI commands, see the [CLI Usage] page or run `heroku
app to the local Git repository (to save having to pass `--app` each time) is not helpful.
Instead, we recommend adding aliases similar to the following to your bash profile:
```bash
alias thd='HEROKU_APP=treeherder-prototype heroku'
alias ths='HEROKU_APP=treeherder-stage heroku'
alias thp='HEROKU_APP=treeherder-prod heroku'
```
```bash
alias thd='HEROKU_APP=treeherder-prototype heroku'
alias ths='HEROKU_APP=treeherder-stage heroku'
alias thp='HEROKU_APP=treeherder-prod heroku'
```
This allows commands to be run against a specific app with minimal typing:
This allows commands to be run against a specific app with minimal typing:
```bash
$ ths config
```
```bash
ths config
```
### Deploying Treeherder
@ -89,14 +89,14 @@ activity can also be seen on the "activity" tab in the Heroku dashboard for each
!!! tip
To simplify pushing latest `master` to the `production` branch, use this bash alias:
```bash
# Replace `origin` with the remote name of the upstream Treeherder repository, if different.
alias deploy='git fetch --all --prune && git push origin remotes/origin/master:production'
```
```bash
# Replace `origin` with the remote name of the upstream Treeherder repository, if different.
alias deploy='git fetch --all --prune && git push origin remotes/origin/master:production'
```
It pushes directly from the `remotes/origin/master` Git metadata branch, meaning the
command works even when the local `master` branch isn't up to date and does not disturb
the locally checked out branch or working directory.
It pushes directly from the `remotes/origin/master` Git metadata branch, meaning the
command works even when the local `master` branch isn't up to date and does not disturb
the locally checked out branch or working directory.
<!-- prettier-ignore -->
!!! warning
@ -151,13 +151,13 @@ spun up for the duration of the command and then destroyed after.
For example to start an interactive bash shell on stage:
```bash
$ heroku run --app treeherder-stage -- bash
heroku run --app treeherder-stage -- bash
```
Or to run a detached Django management command against prod using a larger dyno size:
```bash
$ heroku run:detached --app treeherder-prod --size=standard-2x -- ./manage.py ...
heroku run:detached --app treeherder-prod --size=standard-2x -- ./manage.py ...
```
[one-off dynos]: https://devcenter.heroku.com/articles/one-off-dynos
@ -203,11 +203,11 @@ after logging in with the account ID `moz-devservices` and then [your IAM userna
<!-- prettier-ignore -->
!!! note
For the `treeherder-prod` and `treeherder-stage` Heroku apps, their RDS instances have the
same name as the app. However for `treeherder-prototype` the RDS instance is instead called
`treeherder-dev`.
There is also a read-only replica of production, named `treeherder-prod-ro`.
For the `treeherder-prod` and `treeherder-stage` Heroku apps, their RDS instances have the
same name as the app. However for `treeherder-prototype` the RDS instance is instead called
`treeherder-dev`.
There is also a read-only replica of production, named
`treeherder-prod-ro`.
### Connecting to RDS instances

Просмотреть файл

@ -135,8 +135,8 @@ Here is a set of example parameters that could be used to run it:
You can use the handy Pulse Inspector to view messages in your exchange to
test that they are arriving at Pulse the way you expect. Each exchange has its
own inspector that can be accessed like so: <rootUrl>/pulse-messages/
ex: https://community-tc.services.mozilla.com/pulse-messages/
own inspector that can be accessed like so: `<rootUrl>/pulse-messages/`
ex: <https://community-tc.services.mozilla.com/pulse-messages/>
[pulse guardian]: https://pulseguardian.mozilla.org/whats_pulse
[yml schema]: https://github.com/mozilla/treeherder/blob/master/schemas/pulse-job.yml

Просмотреть файл

@ -1,17 +1,17 @@
# Requirements
This is a directory of requirements files. They are maintained using `pip-tools`:
This is a directory of requirements files. They are maintained using `pip-tools`:
* `*.in` - for developers to enter required packages
* `*.in` - for developers to enter required packages
* `*.txt` - autogenerated by `pip-tools` hash-lock the specific versions for production
## Upgrading
When you want to upgrade (be sure to run from main treeherder directory, this this directory)
When you want to upgrade (be sure to run from main treeherder directory, than this directory)
pip-compile --upgrade --generate-hashes --output-file requirements/common.txt requirements/common.in
pip-compile --upgrade --generate-hashes --output-file requirements/dev.txt requirements/dev.in
pip-compile --upgrade --generate-hashes --output-file requirements/docs.txt requirements/docs.in
> [see pip-tools for more information](https://pypi.org/project/pip-tools/)
> [see pip-tools for more information](https://pypi.org/project/pip-tools/)

Просмотреть файл

@ -9,6 +9,7 @@ responses==0.10.9
django-extensions==2.2.8
pytest-selenium==1.17.0
PyPOM==2.2.0
pre-commit==2.2.0
#Required by isort
seed-isort-config

Просмотреть файл

@ -4,6 +4,10 @@
#
# pip-compile --generate-hashes --output-file=requirements/dev.txt requirements/dev.in
#
appdirs==1.4.3 \
--hash=sha256:9e5896d1372858f8dd3344faf4e5014d21849c756c8d5701f78f8a103b372d92 \
--hash=sha256:d8b24664561d0d34ddfaec54636d502d7cea6e29c3eaf68f3df6180863e2166e \
# via virtualenv
asgiref==3.2.7 \
--hash=sha256:8036f90603c54e93521e5777b2b9a39ba1bad05773fcf2d208f0299d1df58ce5 \
--hash=sha256:9ca8b952a0a9afa61d30aa6d3d9b570bb3fd6bafcf7ec9e6bed43b936133db1c \
@ -28,6 +32,10 @@ certifi==2019.11.28 \
--hash=sha256:017c25db2a153ce562900032d5bc68e9f191e44e9a0f762f373977de9df1fbb3 \
--hash=sha256:25b64c7da4cd7479594d035c08c2d809eb4aab3a26e5a990ea98cc450c320f1f \
# via requests
cfgv==3.1.0 \
--hash=sha256:1ccf53320421aeeb915275a196e23b3b8ae87dea8ac6698b1638001d4a486d53 \
--hash=sha256:c8e8f552ffcc6194f4e18dd4f68d9aef0c0d58ae7e7be8c82bee3c5e9edfa513 \
# via pre-commit
chardet==3.0.4 \
--hash=sha256:84ab92ed1c4d4f16916e05906b6b75a6c0fb5db821cc65e70cbd64a3e2a5eaae \
--hash=sha256:fc323ffcaeaed0e0a02bf4d117757b98aed530d9ed4531e3e15460124c106691 \
@ -73,6 +81,9 @@ coverage==5.0.3 \
--hash=sha256:ea9525e0fef2de9208250d6c5aeeee0138921057cd67fcef90fbed49c4d62d37 \
--hash=sha256:fca1669d464f0c9831fd10be2eef6b86f5ebd76c724d1e0706ebdff86bb4adf0 \
# via -r requirements/dev.in, pytest-cov
distlib==0.3.0 \
--hash=sha256:2e166e231a26b36d6dfe35a48c4464346620f8645ed0ace01ee31822b288de21 \
# via virtualenv
django-debug-toolbar==2.2 \
--hash=sha256:eabbefe89881bbe4ca7c980ff102e3c35c8e8ad6eb725041f538988f2f39a943 \
--hash=sha256:ff94725e7aae74b133d0599b9bf89bd4eb8f5d2c964106e61d11750228c8774c \
@ -89,6 +100,10 @@ entrypoints==0.3 \
--hash=sha256:589f874b313739ad35be6e0cd7efde2a4e9b6fea91edcc34e58ecbb8dbe56d19 \
--hash=sha256:c70dd71abe5a8c85e55e12c19bd91ccfeec11a6e99044204511f9ed547d48451 \
# via -r requirements/dev.in, flake8
filelock==3.0.12 \
--hash=sha256:18d82244ee114f543149c66a6e0c14e9c4f8a1044b5cdaadd0f82159d6a6ff59 \
--hash=sha256:929b7d63ec5b7d6b71b0fa5ac14e030b3f70b75747cef1b10da9b879fef15836 \
# via virtualenv
flake8==3.7.9 \
--hash=sha256:45681a117ecc81e870cbf1262835ae4af5e7a8b08e40b944a8a6e6b895914cfb \
--hash=sha256:49356e766643ad15072a789a20915d3c91dc89fd313ccd71802303fd67e4deca \
@ -97,6 +112,10 @@ funcsigs==1.0.2 \
--hash=sha256:330cc27ccbf7f1e992e69fef78261dc7c6569012cf397db8d3de0234e6c937ca \
--hash=sha256:a7bb0f2cf3a3fd1ab2732cb49eba4252c2af4240442415b4abce3b87022a8f50 \
# via -r requirements/dev.in
identify==1.4.14 \
--hash=sha256:2bb8760d97d8df4408f4e805883dad26a2d076f04be92a10a3e43f09c6060742 \
--hash=sha256:faffea0fd8ec86bb146ac538ac350ed0c73908326426d387eded0bcc9d077522 \
# via pre-commit
idna==2.9 \
--hash=sha256:7588d1c14ae4c77d74036e8c22ff447b26d0fde8f007354fd48a7814db15b7cb \
--hash=sha256:a068a21ceac8a4d63dbfd964670474107f541babbd2250d61922f029858365fa \
@ -104,7 +123,7 @@ idna==2.9 \
importlib-metadata==1.5.0 \
--hash=sha256:06f5b3a99029c7134207dd882428a66992a9de2bef7c2b699b5641f9886c3302 \
--hash=sha256:b97607a1a18a5100839aec1dc26a1ea17ee0d93b20b0f008d80a5a050afb200b \
# via -r requirements/dev.in, pluggy, pytest
# via -r requirements/dev.in, pluggy, pre-commit, pytest, virtualenv
isort==4.3.21 \
--hash=sha256:54da7e92468955c4fceacd0c86bd0ec997b0e1ee80d97f67c35a78b719dccab1 \
--hash=sha256:6e811fcb295968434526407adb8796944f1988c5b65e8139058f2014cbe100fd \
@ -121,6 +140,9 @@ more-itertools==8.2.0 \
--hash=sha256:5dd8bcf33e5f9513ffa06d5ad33d78f31e1931ac9a18f33d37e77a180d393a7c \
--hash=sha256:b1ddb932186d8a6ac451e1d95844b382f55e12686d51ca0c68b6f61f2ab7a507 \
# via -r requirements/dev.in, pytest
nodeenv==1.3.5 \
--hash=sha256:5b2438f2e42af54ca968dd1b374d14a1194848955187b0e5e4be1f73813a5212 \
# via pre-commit
packaging==20.1 \
--hash=sha256:170748228214b70b672c581a3dd610ee51f733018650740e98c7df862a583f73 \
--hash=sha256:e665345f9eef0c621aa0bf2f8d78cf6d21904eef16a93f020240b704a57f1334 \
@ -137,6 +159,10 @@ pluggy==0.13.1 \
--hash=sha256:15b2acde666561e1298d71b523007ed7364de07029219b604cf808bfa1c765b0 \
--hash=sha256:966c145cd83c96502c3c3868f50408687b38434af77734af1e9ca461a4081d2d \
# via -r requirements/dev.in, pypom, pytest
pre-commit==2.2.0 \
--hash=sha256:487c675916e6f99d355ec5595ad77b325689d423ef4839db1ed2f02f639c9522 \
--hash=sha256:c0aa11bce04a7b46c5544723aedf4e81a4d5f64ad1205a30a9ea12d5e81969e1 \
# via -r requirements/dev.in
py==1.8.1 \
--hash=sha256:5e27081401262157467ad6e7f851b7aa402c5852dbcb3dae06768434de5752aa \
--hash=sha256:c20fdd83a5dbc0af9efd622bee9a5564e278f6380fffcacc43ba6f43db2813b0 \
@ -197,6 +223,19 @@ pytz==2019.3 \
--hash=sha256:1c557d7d0e871de1f5ccd5833f60fb2550652da6be2693c1e02300743d21500d \
--hash=sha256:b02c06db6cf09c12dd25137e563b31700d3b80fcc4ad23abb7a315f2789819be \
# via django
pyyaml==5.3.1 \
--hash=sha256:06a0d7ba600ce0b2d2fe2e78453a470b5a6e000a985dd4a4e54e436cc36b0e97 \
--hash=sha256:240097ff019d7c70a4922b6869d8a86407758333f02203e0fc6ff79c5dcede76 \
--hash=sha256:4f4b913ca1a7319b33cfb1369e91e50354d6f07a135f3b901aca02aa95940bd2 \
--hash=sha256:69f00dca373f240f842b2931fb2c7e14ddbacd1397d57157a9b005a6a9942648 \
--hash=sha256:73f099454b799e05e5ab51423c7bcf361c58d3206fa7b0d555426b1f4d9a3eaf \
--hash=sha256:74809a57b329d6cc0fdccee6318f44b9b8649961fa73144a98735b0aaf029f1f \
--hash=sha256:7739fc0fa8205b3ee8808aea45e968bc90082c10aef6ea95e855e10abf4a37b2 \
--hash=sha256:95f71d2af0ff4227885f7a6605c37fd53d3a106fcab511b8860ecca9fcf400ee \
--hash=sha256:b8eac752c5e14d3eca0e6dd9199cd627518cb5ec06add0de9d32baeee6fe645d \
--hash=sha256:cc8955cfbfc7a115fa81d85284ee61147059a753344bc51098f3ccd69b0d7e0c \
--hash=sha256:d13155f591e6fcc1ec3b30685d50bf0711574e2c0dfffd7644babf8b5102ca1a \
# via pre-commit
requests==2.23.0 \
--hash=sha256:43999036bfa82904b6af1d99e4882b560e5e2c68e5c4b0aa03b655f3d7d73fee \
--hash=sha256:b3f43d496c6daba4493e7c431722aeb7dbc6288f52a6e04e7b6023b0247817e6 \
@ -216,15 +255,23 @@ selenium==3.141.0 \
six==1.14.0 \
--hash=sha256:236bdbdce46e6e6a3d61a337c0f8b763ca1e8717c03b369e87a7ec7ce1319c0a \
--hash=sha256:8f3cd2e254d8f793e7f3d6d9df77b92252b52637291d0f0da013c76ea2724b6c \
# via django-extensions, packaging, pip-tools, responses
# via django-extensions, packaging, pip-tools, responses, virtualenv
sqlparse==0.3.1 \
--hash=sha256:022fb9c87b524d1f7862b3037e541f68597a730a8843245c349fc93e1643dc4e \
--hash=sha256:e162203737712307dfe78860cc56c8da8a852ab2ee33750e33aeadf38d12c548 \
# via django, django-debug-toolbar
toml==0.10.0 \
--hash=sha256:229f81c57791a41d65e399fc06bf0848bab550a9dfd5ed66df18ce5f05e73d5c \
--hash=sha256:235682dd292d5899d361a811df37e04a8828a5b1da3115886b73cf81ebc9100e \
# via pre-commit
urllib3==1.25.8 \
--hash=sha256:2f3db8b19923a873b3e5256dc9c2dedfa883e33d87c690d9c7913e1f40673cdc \
--hash=sha256:87716c2d2a7121198ebcb7ce7cccf6ce5e9ba539041cfbaeecfb641dc0bf6acc \
# via requests, selenium
virtualenv==20.0.16 \
--hash=sha256:6ea131d41c477f6c4b7863948a9a54f7fa196854dbef73efbdff32b509f4d8bf \
--hash=sha256:94f647e12d1e6ced2541b93215e51752aecbd1bbb18eb1816e2867f7532b1fe1 \
# via pre-commit
wcwidth==0.1.8 \
--hash=sha256:8fd29383f539be45b20bd4df0dc29c20ba48654a41e661925e612311e9f3c603 \
--hash=sha256:f28b3e8a6483e5d49e7f8949ac1a78314e740333ae305b4ba5defd3e74fb37a8 \

Просмотреть файл

@ -73,7 +73,7 @@ If you made some changes, and want to submit a pull request; run the `./runtests
After `docker-compose up`, you may spin up any number of `backend` containers. You may want to run ingestion tasks, go exploring, or run the tests.
docker-compose exec backend bash
docker-compose has three execution modes
* `exec` - run just the service, and assume the others are running

Просмотреть файл

@ -1,14 +1,12 @@
# Extracting to BigQuery
## Run in Heroku
### Set Environment variables
The Heroku ***Config Vars*** are not delivered to Python environment variables; there is an obfuscation step which changes the characters provided.
The Heroku ***Config Vars*** are not delivered to Python environment variables; there is an obfuscation step which changes the characters provided.
**Some escaping examples**
### Some escaping examples**
| Heroku Config Var | Realized | JSON String* |
| ----------------- | ----------- | -------------------- |
@ -17,11 +15,11 @@ The Heroku ***Config Vars*** are not delivered to Python environment variables;
| `"\n"` | | `"\n"` |
| `"\""` | `"` | `"\""` |
| `"\"\n"` | `"` | `"\"\n"` |
| `"\"\\n\""` | `"\n"` | `"\"\\n\""` |
| `"\"\\n\""` | `"\n"` | `"\"\\n\""` |
***Note:** The JSON String is the JSON encoding of the realized string, for clarity.*
**Some not-escaping examples**
### Some not-escaping examples**
| Heroku Config Var | Realized | JSON String* |
| ----------------- | ----------- | -------------------- |
@ -30,15 +28,14 @@ The Heroku ***Config Vars*** are not delivered to Python environment variables;
| `\"\\n\"` | `\"\\n\"` | `"\\\"\\\\n\\\""` |
| `\n\"\\n\"` | `\n\"\\n\"` | `"\\n\\\"\\\\n\\\""`|
In general, basic escaping works with or without quotes. But if you provide an **invalid** escape sequence, then escaping is disabled; If you try to escape a quote outside quotes, or escape a character that does not require escaping, then the whole string is treated as literal.
### Setup Schedule Job
Heroku has an **Heroku Scheduler Addon** which will create a new machine to execute management commands. You can setup an hourly (or daily) job with the following command:
```
```python
newrelic-admin run-program ./manage.py extract_jobs
```
The `newrelic-admin run-program` prefix ensures NewRelic captures and reports the output.
The `newrelic-admin run-program` prefix ensures NewRelic captures and reports the output.