Add markdownlint pre-commit hook and fixed docs

We also add it as part of the Travis execution.

.markdownlint.json

@@ -0,0 +1,4 @@
+{
+  "default": true,
+  "MD013": false
+}

.markdownlintignore

@@ -0,0 +1 @@
+docs/index.md

.pre-commit-config.yaml

@@ -20,3 +20,7 @@ repos:
     rev: 1.19.1
     hooks:
       - id: prettier
+  - repo: https://github.com/igorshubovych/markdownlint-cli
+    rev: v0.22.0
+    hooks:
+      - id: markdownlint

.travis.yml

@@ -66,6 +66,7 @@ jobs:
         - pip install -r requirements/dev.txt
         - pip install -r requirements/common.txt
       script:
+        - pre-commit run --all-files markdownlint
         - ./runchecks.sh
         - ./manage.py check
         # Several security features in settings.py (eg setting HSTS headers) are conditional on

README.md

@@ -6,21 +6,21 @@
 [![Node devDependencies Status](https://david-dm.org/mozilla/treeherder/dev-status.svg)](https://david-dm.org/mozilla/treeherder?type=dev)
 [![Documentation Status](https://readthedocs.org/projects/treeherder/badge/?version=latest)](https://treeherder.readthedocs.io/?badge=latest)
 
-#### Description
+## Description
 
 [Treeherder](https://treeherder.mozilla.org) is a reporting dashboard for Mozilla checkins. It allows users to see the results of automatic builds and their respective tests. The Treeherder service manages the etl layer for data ingestion, web services, and the data model behind Treeherder.
 
-#### Instances
+## Instances
 
 Treeherder exists on two instances: [staging](https://treeherder.allizom.org) for pre-deployment validation, and [production](https://treeherder.mozilla.org) for actual use.
 
-#### Installation
+## Installation
 
 The steps to run Treeherder are provided [here](https://treeherder.readthedocs.io/installation.html).
 
 The steps to run only the UI are provided [here](https://treeherder.readthedocs.io/installation.html#ui-development).
 
-#### Links
+## Links
 
 Visit our project tracking Wiki [here](https://wiki.mozilla.org/EngineeringProductivity/Projects/Treeherder).
 
@@ -28,8 +28,7 @@ For other setup and configuration, visit our readthedocs page [here](https://t
 
 File any bugs you may encounter [here](https://bugzilla.mozilla.org/enter_bug.cgi?product=Tree+Management&component=Treeherder).
 
-
+## Contributing
-#### Contributing
 
 Everyone is welcome to contribute!
 
@@ -41,7 +40,8 @@ After adressing the issue, make sure [every test passes](https://treeherder.read
 
 We also recommend setting an `upstream` remote that points to the [Mozilla's Github repo](https://github.com/mozilla/treeherder.git), in addition to `origin` that points to your fork. You should then frequently use `git rebase upstream` rather than merging from your fork to keep your branch current. There are less conflicts this way and the git history is cleaner.
 
-##### Sending a Pull Request
+## Sending a Pull Request
+
 We receive contributions from both Bugzilla and Github. We have some specifications to keep track of them:
 
 1. If your bug comes from **[Bugzilla](https://bugzilla.mozilla.org/query.cgi?query_format=advanced&product=Tree+Management&f1=component&o1=substring&v1=Treeherder&resolution=---)**
@@ -55,7 +55,7 @@ We receive contributions from both Bugzilla and Github. We have some specificati
 2. If your bug comes from **Github**
 
     In the **description** of the pull request, please mention the **issue number**. That can be done by typing #[issue's number].
-    
+
     For example: "This pull request fixes #5135".
-    
+
     Github automatically links both issue and pull request to one another.

docs/accessing_data.md

@@ -120,7 +120,7 @@ See the [getting started with ActiveData] guide for more details.
 
 ## Direct database access
 
-If the use-cases above aren't sufficient or you're working on a fullstack Perfherder bug, 
+If the use-cases above aren't sufficient or you're working on a fullstack Perfherder bug,
 we can provide read-only access to Treeherder's production MySQL RDS replica.
 Please [file an infrastructure bug] requesting that someone from the
 Treeherder team [grant access to the read-only replica].

docs/common_tasks.md

@@ -18,6 +18,7 @@ Or if you would rather not use Docker, instead use poetry,run:
 % poetry install
 % poetry run mkdocs serve
 ```
+
 **Note** - On Windows you might need to fallback ```python -m venv venv``` or ```virtualenv``` to manage your virtualenv if ```poetry``` does not work for you.
 
 In either case, the docs will then be available at: <http://localhost:8000>

docs/infrastructure/administration.md

@@ -35,7 +35,7 @@ to perform tasks using the [Heroku CLI]. After installing it, run `heroku login`
 Commands can then be run against a particular app like so:
 
 ```bash
-$ heroku config --app treeherder-stage
+heroku config --app treeherder-stage
 ```
 
 For the list of available CLI commands, see the [CLI Usage] page or run `heroku help`.
@@ -48,17 +48,17 @@ For the list of available CLI commands, see the [CLI Usage] page or run `heroku
     app to the local Git repository (to save having to pass `--app` each time) is not helpful.
     Instead, we recommend adding aliases similar to the following to your bash profile:
 
-    ```bash
+```bash
-    alias thd='HEROKU_APP=treeherder-prototype heroku'
+alias thd='HEROKU_APP=treeherder-prototype heroku'
-    alias ths='HEROKU_APP=treeherder-stage heroku'
+alias ths='HEROKU_APP=treeherder-stage heroku'
-    alias thp='HEROKU_APP=treeherder-prod heroku'
+alias thp='HEROKU_APP=treeherder-prod heroku'
-    ```
+```  
 
-    This allows commands to be run against a specific app with minimal typing:
+This allows commands to be run against a specific app with minimal typing:
 
-    ```bash
+```bash
-    $ ths config
+ths config
-    ```
+```
 
 ### Deploying Treeherder
 
@@ -89,14 +89,14 @@ activity can also be seen on the "activity" tab in the Heroku dashboard for each
 !!! tip
     To simplify pushing latest `master` to the `production` branch, use this bash alias:
 
-    ```bash
+```bash
-    # Replace `origin` with the remote name of the upstream Treeherder repository, if different.
+# Replace `origin` with the remote name of the upstream Treeherder repository, if different.
-    alias deploy='git fetch --all --prune && git push origin remotes/origin/master:production'
+alias deploy='git fetch --all --prune && git push origin remotes/origin/master:production'
-    ```
+```
 
-    It pushes directly from the `remotes/origin/master` Git metadata branch, meaning the
+It pushes directly from the `remotes/origin/master` Git metadata branch, meaning the
-    command works even when the local `master` branch isn't up to date and does not disturb
+command works even when the local `master` branch isn't up to date and does not disturb
-    the locally checked out branch or working directory.
+the locally checked out branch or working directory.
 
 <!-- prettier-ignore -->
 !!! warning
@@ -151,13 +151,13 @@ spun up for the duration of the command and then destroyed after.
 For example to start an interactive bash shell on stage:
 
 ```bash
-$ heroku run --app treeherder-stage -- bash
+heroku run --app treeherder-stage -- bash
 ```
 
 Or to run a detached Django management command against prod using a larger dyno size:
 
 ```bash
-$ heroku run:detached --app treeherder-prod --size=standard-2x -- ./manage.py ...
+heroku run:detached --app treeherder-prod --size=standard-2x -- ./manage.py ...
 ```
 
 [one-off dynos]: https://devcenter.heroku.com/articles/one-off-dynos
@@ -203,11 +203,11 @@ after logging in with the account ID `moz-devservices` and then [your IAM userna
 
 <!-- prettier-ignore -->
 !!! note
-    For the `treeherder-prod` and `treeherder-stage` Heroku apps, their RDS instances have the
+For the `treeherder-prod` and `treeherder-stage` Heroku apps, their RDS instances have the
-    same name as the app. However for `treeherder-prototype` the RDS instance is instead called
+same name as the app. However for `treeherder-prototype` the RDS instance is instead called
-    `treeherder-dev`.
+`treeherder-dev`.
-
+There is also a read-only replica of production, named
-    There is also a read-only replica of production, named `treeherder-prod-ro`.
+`treeherder-prod-ro`.
 
 ### Connecting to RDS instances
 

docs/pulseload.md

@@ -135,8 +135,8 @@ Here is a set of example parameters that could be used to run it:
 
 You can use the handy Pulse Inspector to view messages in your exchange to
 test that they are arriving at Pulse the way you expect. Each exchange has its
-own inspector that can be accessed like so: <rootUrl>/pulse-messages/
+own inspector that can be accessed like so: `<rootUrl>/pulse-messages/`
-ex: https://community-tc.services.mozilla.com/pulse-messages/
+ex: <https://community-tc.services.mozilla.com/pulse-messages/>
 
 [pulse guardian]: https://pulseguardian.mozilla.org/whats_pulse
 [yml schema]: https://github.com/mozilla/treeherder/blob/master/schemas/pulse-job.yml

requirements/README.md

@@ -1,17 +1,17 @@
 
 # Requirements
 
-This is a directory of requirements files. They are maintained using `pip-tools`: 
+This is a directory of requirements files. They are maintained using `pip-tools`:
 
-* `*.in` - for developers to enter required packages 
+* `*.in` - for developers to enter required packages
 * `*.txt` - autogenerated by `pip-tools` hash-lock the specific versions for production
 
 ## Upgrading
 
-When you want to upgrade (be sure to run from main treeherder directory, this this directory)
+When you want to upgrade (be sure to run from main treeherder directory, than this directory)
 
     pip-compile --upgrade --generate-hashes --output-file requirements/common.txt requirements/common.in
     pip-compile --upgrade --generate-hashes --output-file requirements/dev.txt requirements/dev.in
     pip-compile --upgrade --generate-hashes --output-file requirements/docs.txt requirements/docs.in
 
-> [see pip-tools for more information](https://pypi.org/project/pip-tools/)
+> [see pip-tools for more information](https://pypi.org/project/pip-tools/)

requirements/dev.in

@@ -9,6 +9,7 @@ responses==0.10.9
 django-extensions==2.2.8
 pytest-selenium==1.17.0
 PyPOM==2.2.0
+pre-commit==2.2.0
 
 #Required by isort
 seed-isort-config

requirements/dev.txt

@@ -4,6 +4,10 @@
 #
 #    pip-compile --generate-hashes --output-file=requirements/dev.txt requirements/dev.in
 #
+appdirs==1.4.3 \
+    --hash=sha256:9e5896d1372858f8dd3344faf4e5014d21849c756c8d5701f78f8a103b372d92 \
+    --hash=sha256:d8b24664561d0d34ddfaec54636d502d7cea6e29c3eaf68f3df6180863e2166e \
+    # via virtualenv
 asgiref==3.2.7 \
     --hash=sha256:8036f90603c54e93521e5777b2b9a39ba1bad05773fcf2d208f0299d1df58ce5 \
     --hash=sha256:9ca8b952a0a9afa61d30aa6d3d9b570bb3fd6bafcf7ec9e6bed43b936133db1c \
@@ -28,6 +32,10 @@ certifi==2019.11.28 \
     --hash=sha256:017c25db2a153ce562900032d5bc68e9f191e44e9a0f762f373977de9df1fbb3 \
     --hash=sha256:25b64c7da4cd7479594d035c08c2d809eb4aab3a26e5a990ea98cc450c320f1f \
     # via requests
+cfgv==3.1.0 \
+    --hash=sha256:1ccf53320421aeeb915275a196e23b3b8ae87dea8ac6698b1638001d4a486d53 \
+    --hash=sha256:c8e8f552ffcc6194f4e18dd4f68d9aef0c0d58ae7e7be8c82bee3c5e9edfa513 \
+    # via pre-commit
 chardet==3.0.4 \
     --hash=sha256:84ab92ed1c4d4f16916e05906b6b75a6c0fb5db821cc65e70cbd64a3e2a5eaae \
     --hash=sha256:fc323ffcaeaed0e0a02bf4d117757b98aed530d9ed4531e3e15460124c106691 \
@@ -73,6 +81,9 @@ coverage==5.0.3 \
     --hash=sha256:ea9525e0fef2de9208250d6c5aeeee0138921057cd67fcef90fbed49c4d62d37 \
     --hash=sha256:fca1669d464f0c9831fd10be2eef6b86f5ebd76c724d1e0706ebdff86bb4adf0 \
     # via -r requirements/dev.in, pytest-cov
+distlib==0.3.0 \
+    --hash=sha256:2e166e231a26b36d6dfe35a48c4464346620f8645ed0ace01ee31822b288de21 \
+    # via virtualenv
 django-debug-toolbar==2.2 \
     --hash=sha256:eabbefe89881bbe4ca7c980ff102e3c35c8e8ad6eb725041f538988f2f39a943 \
     --hash=sha256:ff94725e7aae74b133d0599b9bf89bd4eb8f5d2c964106e61d11750228c8774c \
@@ -89,6 +100,10 @@ entrypoints==0.3 \
     --hash=sha256:589f874b313739ad35be6e0cd7efde2a4e9b6fea91edcc34e58ecbb8dbe56d19 \
     --hash=sha256:c70dd71abe5a8c85e55e12c19bd91ccfeec11a6e99044204511f9ed547d48451 \
     # via -r requirements/dev.in, flake8
+filelock==3.0.12 \
+    --hash=sha256:18d82244ee114f543149c66a6e0c14e9c4f8a1044b5cdaadd0f82159d6a6ff59 \
+    --hash=sha256:929b7d63ec5b7d6b71b0fa5ac14e030b3f70b75747cef1b10da9b879fef15836 \
+    # via virtualenv
 flake8==3.7.9 \
     --hash=sha256:45681a117ecc81e870cbf1262835ae4af5e7a8b08e40b944a8a6e6b895914cfb \
     --hash=sha256:49356e766643ad15072a789a20915d3c91dc89fd313ccd71802303fd67e4deca \
@@ -97,6 +112,10 @@ funcsigs==1.0.2 \
     --hash=sha256:330cc27ccbf7f1e992e69fef78261dc7c6569012cf397db8d3de0234e6c937ca \
     --hash=sha256:a7bb0f2cf3a3fd1ab2732cb49eba4252c2af4240442415b4abce3b87022a8f50 \
     # via -r requirements/dev.in
+identify==1.4.14 \
+    --hash=sha256:2bb8760d97d8df4408f4e805883dad26a2d076f04be92a10a3e43f09c6060742 \
+    --hash=sha256:faffea0fd8ec86bb146ac538ac350ed0c73908326426d387eded0bcc9d077522 \
+    # via pre-commit
 idna==2.9 \
     --hash=sha256:7588d1c14ae4c77d74036e8c22ff447b26d0fde8f007354fd48a7814db15b7cb \
     --hash=sha256:a068a21ceac8a4d63dbfd964670474107f541babbd2250d61922f029858365fa \
@@ -104,7 +123,7 @@ idna==2.9 \
 importlib-metadata==1.5.0 \
     --hash=sha256:06f5b3a99029c7134207dd882428a66992a9de2bef7c2b699b5641f9886c3302 \
     --hash=sha256:b97607a1a18a5100839aec1dc26a1ea17ee0d93b20b0f008d80a5a050afb200b \
-    # via -r requirements/dev.in, pluggy, pytest
+    # via -r requirements/dev.in, pluggy, pre-commit, pytest, virtualenv
 isort==4.3.21 \
     --hash=sha256:54da7e92468955c4fceacd0c86bd0ec997b0e1ee80d97f67c35a78b719dccab1 \
     --hash=sha256:6e811fcb295968434526407adb8796944f1988c5b65e8139058f2014cbe100fd \
@@ -121,6 +140,9 @@ more-itertools==8.2.0 \
     --hash=sha256:5dd8bcf33e5f9513ffa06d5ad33d78f31e1931ac9a18f33d37e77a180d393a7c \
     --hash=sha256:b1ddb932186d8a6ac451e1d95844b382f55e12686d51ca0c68b6f61f2ab7a507 \
     # via -r requirements/dev.in, pytest
+nodeenv==1.3.5 \
+    --hash=sha256:5b2438f2e42af54ca968dd1b374d14a1194848955187b0e5e4be1f73813a5212 \
+    # via pre-commit
 packaging==20.1 \
     --hash=sha256:170748228214b70b672c581a3dd610ee51f733018650740e98c7df862a583f73 \
     --hash=sha256:e665345f9eef0c621aa0bf2f8d78cf6d21904eef16a93f020240b704a57f1334 \
@@ -137,6 +159,10 @@ pluggy==0.13.1 \
     --hash=sha256:15b2acde666561e1298d71b523007ed7364de07029219b604cf808bfa1c765b0 \
     --hash=sha256:966c145cd83c96502c3c3868f50408687b38434af77734af1e9ca461a4081d2d \
     # via -r requirements/dev.in, pypom, pytest
+pre-commit==2.2.0 \
+    --hash=sha256:487c675916e6f99d355ec5595ad77b325689d423ef4839db1ed2f02f639c9522 \
+    --hash=sha256:c0aa11bce04a7b46c5544723aedf4e81a4d5f64ad1205a30a9ea12d5e81969e1 \
+    # via -r requirements/dev.in
 py==1.8.1 \
     --hash=sha256:5e27081401262157467ad6e7f851b7aa402c5852dbcb3dae06768434de5752aa \
     --hash=sha256:c20fdd83a5dbc0af9efd622bee9a5564e278f6380fffcacc43ba6f43db2813b0 \
@@ -197,6 +223,19 @@ pytz==2019.3 \
     --hash=sha256:1c557d7d0e871de1f5ccd5833f60fb2550652da6be2693c1e02300743d21500d \
     --hash=sha256:b02c06db6cf09c12dd25137e563b31700d3b80fcc4ad23abb7a315f2789819be \
     # via django
+pyyaml==5.3.1 \
+    --hash=sha256:06a0d7ba600ce0b2d2fe2e78453a470b5a6e000a985dd4a4e54e436cc36b0e97 \
+    --hash=sha256:240097ff019d7c70a4922b6869d8a86407758333f02203e0fc6ff79c5dcede76 \
+    --hash=sha256:4f4b913ca1a7319b33cfb1369e91e50354d6f07a135f3b901aca02aa95940bd2 \
+    --hash=sha256:69f00dca373f240f842b2931fb2c7e14ddbacd1397d57157a9b005a6a9942648 \
+    --hash=sha256:73f099454b799e05e5ab51423c7bcf361c58d3206fa7b0d555426b1f4d9a3eaf \
+    --hash=sha256:74809a57b329d6cc0fdccee6318f44b9b8649961fa73144a98735b0aaf029f1f \
+    --hash=sha256:7739fc0fa8205b3ee8808aea45e968bc90082c10aef6ea95e855e10abf4a37b2 \
+    --hash=sha256:95f71d2af0ff4227885f7a6605c37fd53d3a106fcab511b8860ecca9fcf400ee \
+    --hash=sha256:b8eac752c5e14d3eca0e6dd9199cd627518cb5ec06add0de9d32baeee6fe645d \
+    --hash=sha256:cc8955cfbfc7a115fa81d85284ee61147059a753344bc51098f3ccd69b0d7e0c \
+    --hash=sha256:d13155f591e6fcc1ec3b30685d50bf0711574e2c0dfffd7644babf8b5102ca1a \
+    # via pre-commit
 requests==2.23.0 \
     --hash=sha256:43999036bfa82904b6af1d99e4882b560e5e2c68e5c4b0aa03b655f3d7d73fee \
     --hash=sha256:b3f43d496c6daba4493e7c431722aeb7dbc6288f52a6e04e7b6023b0247817e6 \
@@ -216,15 +255,23 @@ selenium==3.141.0 \
 six==1.14.0 \
     --hash=sha256:236bdbdce46e6e6a3d61a337c0f8b763ca1e8717c03b369e87a7ec7ce1319c0a \
     --hash=sha256:8f3cd2e254d8f793e7f3d6d9df77b92252b52637291d0f0da013c76ea2724b6c \
-    # via django-extensions, packaging, pip-tools, responses
+    # via django-extensions, packaging, pip-tools, responses, virtualenv
 sqlparse==0.3.1 \
     --hash=sha256:022fb9c87b524d1f7862b3037e541f68597a730a8843245c349fc93e1643dc4e \
     --hash=sha256:e162203737712307dfe78860cc56c8da8a852ab2ee33750e33aeadf38d12c548 \
     # via django, django-debug-toolbar
+toml==0.10.0 \
+    --hash=sha256:229f81c57791a41d65e399fc06bf0848bab550a9dfd5ed66df18ce5f05e73d5c \
+    --hash=sha256:235682dd292d5899d361a811df37e04a8828a5b1da3115886b73cf81ebc9100e \
+    # via pre-commit
 urllib3==1.25.8 \
     --hash=sha256:2f3db8b19923a873b3e5256dc9c2dedfa883e33d87c690d9c7913e1f40673cdc \
     --hash=sha256:87716c2d2a7121198ebcb7ce7cccf6ce5e9ba539041cfbaeecfb641dc0bf6acc \
     # via requests, selenium
+virtualenv==20.0.16 \
+    --hash=sha256:6ea131d41c477f6c4b7863948a9a54f7fa196854dbef73efbdff32b509f4d8bf \
+    --hash=sha256:94f647e12d1e6ced2541b93215e51752aecbd1bbb18eb1816e2867f7532b1fe1 \
+    # via pre-commit
 wcwidth==0.1.8 \
     --hash=sha256:8fd29383f539be45b20bd4df0dc29c20ba48654a41e661925e612311e9f3c603 \
     --hash=sha256:f28b3e8a6483e5d49e7f8949ac1a78314e740333ae305b4ba5defd3e74fb37a8 \

tests/README.md

@@ -73,7 +73,7 @@ If you made some changes, and want to submit a pull request; run the `./runtests
 After `docker-compose up`, you may spin up any number of `backend` containers. You may want to run ingestion tasks, go exploring, or run the tests.
 
     docker-compose exec backend bash
-    
+
 docker-compose has three execution modes
 
 * `exec` - run just the service, and assume the others are running

treeherder/extract/README.md

@@ -1,14 +1,12 @@
-
 # Extracting to BigQuery
 
 ## Run in Heroku
 
-
 ### Set Environment variables
 
-The Heroku ***Config Vars*** are not delivered to Python environment variables; there is an obfuscation step which changes the characters provided. 
+The Heroku ***Config Vars*** are not delivered to Python environment variables; there is an obfuscation step which changes the characters provided.
 
-**Some escaping examples**
+### Some escaping examples**
 
 | Heroku Config Var |  Realized   | JSON String*         |
 | ----------------- | ----------- | -------------------- |
@@ -17,11 +15,11 @@ The Heroku ***Config Vars*** are not delivered to Python environment variables;
 |      `"\n"`       |             |  `"\n"`              |
 |      `"\""`       | `"`         |  `"\""`              |
 |      `"\"\n"`     | `"`         |  `"\"\n"`            |
-|      `"\"\\n\""`  | `"\n"`      |  `"\"\\n\""`         | 
+|      `"\"\\n\""`  | `"\n"`      |  `"\"\\n\""`         |
 
 ***Note:** The JSON String is the JSON encoding of the realized string, for clarity.*
 
-**Some not-escaping examples**
+### Some not-escaping examples**
 
 | Heroku Config Var |  Realized   | JSON String*         |
 | ----------------- | ----------- | -------------------- |
@@ -30,15 +28,14 @@ The Heroku ***Config Vars*** are not delivered to Python environment variables;
 |      `\"\\n\"`    | `\"\\n\"`   |  `"\\\"\\\\n\\\""`   |  
 |      `\n\"\\n\"`  | `\n\"\\n\"` |  `"\\n\\\"\\\\n\\\""`|
 
-
 In general, basic escaping works with or without quotes. But if you provide an **invalid** escape sequence, then escaping is disabled; If you try to escape a quote outside quotes, or escape a character that does not require escaping, then the whole string is treated as literal.
 
 ### Setup Schedule Job
 
 Heroku has an **Heroku Scheduler Addon** which will create a new machine to execute management commands. You can setup an hourly (or daily) job with the following command:
 
-```
+```python
 newrelic-admin run-program ./manage.py extract_jobs
 ```
 
-The `newrelic-admin run-program` prefix ensures NewRelic captures and reports the output. 
+The `newrelic-admin run-program` prefix ensures NewRelic captures and reports the output.