Add a lot of documentation to cover the whole project (#576)

README.md

@@ -1,12 +1,14 @@
 # Mozilla Code Review
 
-This project has 2 parts:
+The **Code Review Bot** aims to give early feedback to Mozilla developers about their patches. We automate code analyzers and publish detected issues on Phabricator as soon as possible, and for all revisions.
+
+This project has 4 parts:
 
 * `bot` is a Python script running as a Taskcluster task, reporting issues found in analyzer tasks,
-* `frontend` is an administration frontend (in Vue.js) displaying detailed information about current analyses.
+* `backend` is a Django web API used to store issues detected by the bot,
+* `frontend` is an administration frontend (in Vue.js) displaying detailed information about analyses and issues,
+* `events` is a Python distributed application running in Heroku that receives Phabricator notifications and triggers Try pushes,
 
-The analyzer tasks are triggered by [pulselistener](https://github.com/mozilla/release-services/tree/master/src/pulselistener), from the release-services project.
-
-:blue_book: Documentation is available in this repository [in the docs folder](docs/summary.md).
+:blue_book: Documentation is available in this repository [in the docs folder](docs/README.md). A good starting point is the [architecture description](docs/architecture.md).
 
 :loudspeaker: You can contact the code review bot's developers [on Matrix](https://chat.mozilla.org/#/room/#code-review-bot:mozilla.org) or on Slack in #code-review-bot.

bot/README.md

@@ -47,6 +47,8 @@ Supported reporting tools are emails (for admins) and Phabricator.
 
 Each reporter configuration must contain a `reporter` key with a unique name per tool. Each tool has its own configuration requirement.
 
+You can view a [full configuration sample here](/docs/configuration.md).
+
 Phabricator credentials
 -----------------------
 
@@ -91,7 +93,7 @@ common:
     url: https://dev.phabricator.mozilla.com
     api_key: deadbeef123456
 
-code-review-bot:
+bot:
   REPORTERS:
   - reporter: phabricator
 ```

docs/README.md

@@ -4,6 +4,16 @@ This documentation is targeted towards developers of the code review bot itself,
 
 Here is the overall summary of available documentation:
 
+* [System Architecture](architecture.md) (good starting point)
+* [System Configuration](configuration.md)
+* [How to debug the code review bot](debugging.md) and other tips for maintainers
+* [CI/CD Pipeline](ci-cd/README.md) with Taskcluster
+* [Phabricator](phabricator.md) integrations, lessons learned, tips & tricks
+* Projects details:
+    - [backend](projects/backend.md)
+    - [bot](projects/bot.md)
+    - [events](projects/events.md)
+    - [frontend](projects/bot.md)
 * How to add a new analyzer:
     - [How to trigger your task on new diffs](trigger.md)
     - [How to publish issues on Phabricator](publication.md)

docs/analysis_format.md

@@ -37,7 +37,7 @@ Here is an example of an analysis using this format:
 }
 ```
 
-We have built a validation tool, available in this repository as `bot/tools/validator.py`. It has no extra dependencies, and can run using any Python 3 version. You can download it directly on your computer to troubleshoot your payloads:
+We have built a validation tool, available in this repository as `bot/tools/validator.py`. It has no extra dependencies, and can run using any Python version. You can download it directly on your computer to troubleshoot your payloads:
 
 ```
 wget https://raw.githubusercontent.com/mozilla/code-review/master/bot/tools/validator.py

docs/architecture.drawio

@@ -0,0 +1 @@
+<mxfile host="app.diagrams.net" modified="2020-03-30T10:39:56.177Z" agent="5.0 (X11)" etag="V4tiKjywU8QOoRj9M7lB" version="12.9.4" type="device"><diagram id="lIvQNT1Oq-e08xpbe9e6" name="Page-1">7Vtdd5s4EP01fkwPCPP1mOJ4e07bbXqyPWkeBciYDUZeWcR2f/0KEGCBiHFsIEnrhwSNhIxm7tyZEfJEc1a7vwhcL79iH0UToPi7iTabAGBagP1NBftcoKomlwQk9LmsEtyFvxAXKlyahD7aCAMpxhEN16LQw3GMPCrIICF4Kw5b4Ej81jUMUENw58GoKb0PfbrMpRYwK/knFAbL4ptVw857VrAYzFeyWUIfbw9E2s1EcwjGNL9a7RwUpcor9JLfN2/pLR+MoJh2uWHr//i+ib8/mNH0788z/3Oyv/Kv+CxPMEr4gifAiNh8HxeYTcuemu65Koz/Elx0XG0yQ12zAaqx3lWdmSUiTIQ7JkBT2Gc+PxQZQfr/dgldEnqQslv4F7MV5N+dj+DKKx8DEJzEPkoXpbLu7TKk6G4NvbR3yzDIZEu6inh3OtMcrsIoxd8PN4lpkkrDKHLKx9R8iKyFx+QbSvAjOugxPAu5i/IpnhChaNeqf7W0KnMHhFeIkj0bshMRzT2hxMW2gpVayJYHkDK4DHIkB+XMlbHZBbf3CbYHMuPXlB0wba9bF8+dDrrFcOVkpZgdtGJJtKKpek9qMY5r5TQIimBznPnccWRgs4CrGUZXsD1j0qa2x9MmOEYwLXThOIrSpIvavYtc59U47uFH6Wg6s23Z/PfIbaWhXOySuuQYX1ESwjhIW8eQQjCFNMQxa9qKhKZ0ZPnTgZGzE0mIu+lUb3ipIYGV1Rd1aQOFrTacnI7D/L6viHgJw0O6wHtMHlF71Hsp3M7hpjERpokIU60mxIYNj+oFciOrBrKLE9YnxAyVdKYitKMiAERDxzhGNVRwEYzCIOUlj9mYgVb7mCKA5W3RNe9Yhb7fynEVJpU+IVTPJSQQmkoQBPpC0LRhAOSz4oI3MaFLHOAYRjeVtKaqaswXjNfcZv8iSve8UoIJxaJF0S6kPw+uH9KpPui8NdvxmbPGvmjEbLk/q4Fp8+Gwr7ota+0FS6aLeokdmWZwQrhDPJs9UEgC9NyEmhwXBEUsoD6JT3f5FEeW4wycR+v68Txasw1Z5tcbf077TaTn1g37yIKVa+lT/UymAUcT6aHVqb+icDSfpwV9Mxz9AzePXpRs6DOJzXuLSS1IKQtcoxaUmo45aFACxhhR6cWRouCMY5FCNTqGCtUaNVaYo+QEZXzvGt1B2bxFJGRrTx3svJh/eUuao1rS+s39yB5T+6psl+4ye0NtsW2GvHCT7s5cfN+nltgsLA950u3oHhObYhoxWqnq2Bs92ih8yWuoolB6OLjOyNIWSqhp0apzZQvvVkXWpeuqzl5udvRyTZHDZSAvNwf38muGk/2v7unrZgnX6aWXuB083c2R+cUtBdB7DDK8fktoFKZ5bSb3IXn8xqYJaYaJD4ouCkEmfW3kAcQiVG+Shz3kDp426gbMueTRPwlYb4MErD8k8IZJQAVjs4D+PhN1+214r/3He9+y99pje68yOH5uEzdKjyS991qv/lJfxtSDFntgFKbusUjr/JpL68jleXk+2pbXBY5dTC/1CsJxDEP6CiJTyB0iT90jwDmvoRaLBZB7sG+4ht75zMSR0xG26Kua5ATOwKcHxymuxnfWrr4KRn0prR09vfJiz1MUVZWd5f3IEiEU+7KjS5LzTDNIoQs36OQsbc9yLD9733epTG2U48CWoQserUtOq5gSh1aV3lIt0FD673depTMLKF3LrxYYdKaB7NZrQuD+YMAahzHdHMx8mwoqdNlKLV5wdM07jtfF8ewif4IKXeVSzgDcyC9DTwPPS4A6BuDeJN6KU+lt481nh/cEz1FOSryCDEftiLSW8vLcBIc1q19F5casflum3fwP</diagram></mxfile>

docs/architecture.md

@@ -0,0 +1,93 @@
+# Architecture
+
+The code review bot is not a single piece, but a distributed system using several providers.
+
+![](architecture.png)
+
+[Source](architecture.drawio) to use on [draw.io](https://app.diagrams.net/).
+
+## Workflow
+
+1. For every new diff on the [Mozilla Phabricator instance](https://phabricator.services.mozilla.com/), an Harbormaster build is created (more details on the [Phabricator page](phabricator.md)
+2. The build triggers an HTTP request towards the Events system hosted on Heroku.
+3. The Heroku web instance parses the Phabricator payload and stores the relevant information in a Redis database
+4. A pool of Heroku workers polls the Redis database, one of them retrieves the stored build,
+5. That worker then fetches the stack of patches, and tries to apply them locally,
+6. If the stack fails to apply, an error is reported directly on the Phabricator revision,
+7. If the stack applies cleanly, it's pushed on the Try server, exactly like a developer would do,
+8. The Try server creates a decision task on Taskcluster,
+9. The Decision tasks creates followup tasks, with code analyzers, and a custom code-review task.
+10. The code analyzers run on the stack we produced, and list potential issues in a JSON file, that is stored and shared as a Taskcluster artifact,
+11. The final code-review task is triggered once all analyzers complete and just sends a Pulse message.
+12. The pulse messages triggers our own task hosted in a Taskcluster hook (see [bot](projects/bot.md) for more information)
+13. That task retrieves all the analyzers artifacts with issues, filters them, and updates the Phabricator build, reporting potential issues.
+14. That same task also publishes the issues on our backend hosted on Heroku (see [backend](projects/backend.md) for more information).
+
+The whole workflow takes about 12 to 15 minutes on average for a given revision.
+
+## Events
+
+The events system uses both Dyno types available on Heroku:
+
+- a unique **web** Dyno, exposed as https://events.code-review.moz.tools
+- at least one **worker** Dyno, not reachable publicly
+
+The **web** Dyno only stores HTTP requests from Phabricator into the shared Redis Database (also provided by Heroku as an addon).
+
+Each **worker** clones a list of Mozilla repositories at startup (including the huge [mozilla-central](https://hg.mozilla.org/mozilla-central/)), polls the Redis database for new builds to run.
+
+For each build retrieved, the stack of patches is retrieved from the Phabricator API (and not HGMO), then applied locally on the target repository. Once the stack is cleanly applied, a mercurial push to the corresponding try server is done.
+
+Finally, the Phabricator build is updated with the treeherder link, so that developers can view the progress of their review build while it's running (also useful for debugging !).
+
+The list of repositories, try servers and options to run the Events system on Heroku is stored in [the shared configuration on Taskcluster](configuration.md).
+
+## Analyzers
+
+The whole project relies on the code analyzers defined on the target repositories. They can produce issues detected by static analyzers, or formatting issues, with patches to improve the code quality.
+
+We currently support several analyzers used in Mozilla central:
+
+- clang-format
+- clang-tidy
+- coverity
+- infer
+- mozlint (bringing a huge list of analyzers like flake8, eslint, rustfmt, ...)
+
+All those analyzers output a different JSON payload, hence the need for different implementations. But we also have a [default format](analysis_format.md) that is recommended if you want to add a new analyzer.
+
+In an ideal world, we would only support that format, and convert the existing analyzers to output a compatible payload.
+
+## Publication task
+
+You may have noticed the *pulse message dance* in the workflow above. It may seem weird to add a task only sending a Pulse message to trigger another task in the same Taskcluster instance.
+
+This has brought us several advantages:
+- we do not [ever](https://hg.mozilla.org/mozilla-central/log/tip/taskcluster/ci/code-review/kind.yml) need to update the first task, so we do not have a direct dependency on mozilla-central
+- we can update the publication code (the **bot** part) easily, just like any other Taskcluster hook payload
+- we can trigger the hook from different projects, with different rules and workflows
+- we get a trigger history for free thanks to the Taskcluster hook system
+- we can run several versions of the hook in parallel (the testing environment can receive the same pulse messages than production, allowing to run in *shadow mode* - see [debugging](debugging.md) for that)
+
+## Backend
+
+The backend is hosted on a single Heroku web dyno at https://api.code-review.moz.tools and linked to an Heroku Postgres database. This setup would allow us to scale easily if the need arises.
+
+The Django web framework is used to build an efficient REST API (with [django-rest-framework](https://www.django-rest-framework.org/)), based on the official ORM system.
+
+Its main purpose is to store issues reported by the publication task at the end of its run, so that they can be retrieved later on by the frontend, if some developer wants to see the reports for any revision in more details.
+
+For more information, read the [backend](backend.md) documentation.
+
+## Frontend
+
+The frontend is a simple [Vue.js](https://vuejs.org) Single Page Application hosted under an S3+Cloudfront bucket as https://code-review.moz.tools.
+It uses the backend as its main source of information, no authentication is implemented as all information is already public (we do not support security bugs).
+
+With the frontend, you can:
+- browse analysis tasks (also called reports)
+- search them by revision, bugzilla id, name, ...
+- list all the issues detected in a diff
+- compare the evolution of issues in a revision, across diffs
+- view the publication statistics (which analyzers report the most, ...)
+- view issues per analyzers

docs/ci-cd/README.md

@@ -0,0 +1,87 @@
+# CI/CD Pipeline with Taskcluster
+
+The code review bot relies **extensively** on Taskcluster, to run publication tasks, code analysis, but also to build and ship the code-review bot itself.
+
+We use both Mozilla's Taskcluster instances:
+
+- [firefox-ci](https://firefox-ci-tc.services.mozilla.com/) to build and run the bot project
+- [community](https://community-tc.services.mozilla.com/) to build the other projects
+
+We use a pretty simple convention related to the git workflow:
+
+1. Every pull request triggers a pipeline of tasks check unit tests, code format, ultimately building docker images.
+2. The `master` branch must always be *clean* and deployable. That branch is protected.
+3. Tags are only used to represent a release, using semantic versioning (example on [release 1.2.1](https://github.com/mozilla/code-review/releases/tag/1.2.1))
+4. A push on `testing` will run the full pipeline, building Docker images, and deploying them on the different providers for the **testing** environment (Taskcluster & Heroku)
+5. A push on `production` will run the full pipeline, building Docker images, and deploying them on the different providers for the **production** environment (Taskcluster & Heroku)
+
+
+## Firefox-CI instance
+
+![](firefox-ci.png)
+[Graph source code](firefox-ci.mermaid)
+
+The CI/CD workflow is really simple on the firefox-ci instance. On every Github push (branch, pull request, tag), we run the following tasks:
+
+- `check_lint` checks the formatting of all files
+- `bot_check_tests` runs the bot unit tests
+- `bot_build_dind` builds the bot's Docker image using Docker In Docker (dind), stores the image as a public artifact, and the task is always indexed.
+
+That last task is interesting because it allows us to ship immediately updates for the code review bot in the Taskcluster firefox-ci instance without relying on a remote Docker repository: the image is built in Taskcluster and stays there. Other tasks can use our docker image by referencing it through the [Taskcluster task-image](https://firefox-source-docs.mozilla.org/taskcluster/docker-images.html) feature.
+
+You can view that usage in the bot's [production](https://firefox-ci-tc.services.mozilla.com/hooks/project-relman/code-review-production) & [testing](https://firefox-ci-tc.services.mozilla.com/hooks/project-relman/code-review-testing) hooks.
+
+### Routes
+
+On a github pull request, the docker image is indexed as below. `head_rev` is the revision sha hash, `head_branch` is the remote branch name.
+
+```
+index.code-analysis.v2.code-review-pr.revision.${head_rev}
+index.code-analysis.v2.code-review-pr.branch.${head_branch}
+```
+
+On a github branch push (especially for **testing** & **production** branches) as below. `head_rev` is the revision sha hash, `head_branch` is the local branch name.
+
+```
+index.code-analysis.v2.code-review.revision.${head_rev}
+index.code-analysis.v2.code-review.branch.${head_branch}
+```
+
+As mentioned in the previous paragraph, the pre-configured hooks rely on the `index.code-analysis.v2.code-review.branch.testing` & `index.code-analysis.v2.code-review.branch.production` indexes.
+
+### Configuration
+
+The firefox-ci configuration is stored in https://hg.mozilla.org/ci/ci-configuration/
+
+To make changes, lookup the relevant code-review sections in the `grants.yml`, `hooks.yml` and `projects.yml` files. You'll need to submit your patch on Phabricator, linked to a Bugzilla bug filed under **Release Engineering :: Firefox-CI Administration** (example on [Bug 1604227](https://bugzilla.mozilla.org/show_bug.cgi?id=1604227))
+
+You can also ask for help on Matrix #firefox-ci channel.
+
+## Community instance
+
+![](community.png)
+[Graph source code](community.mermaid)
+
+As you can see, it's a bit more complicated on the Community instance. But the goal is similar: we only aim to build Docker images continuously and deploy them on their target when we push on the corresponding deployment branches (testing & production).
+
+On every Github push (branch, pull request, tag), we run the following tasks:
+
+- `check_lint` checks the formatting of all files
+- `backend_check_tests` runs the backend unit tests
+- `events_check_tests` runs the events unit tests
+- `integration_check_tests` runs the integration unit tests
+- `backend_build` builds the backend's Docker image using img and a privileged worker, and stores it as a public artifact
+- `events_build` builds the events's Docker image using img and a privileged worker, and stores it as a public artifact
+- `integration_build` builds the integration's Docker image using img and a privileged worker, and stores it as a public artifact
+- `frontend_build` builds the frontend static files and stores them as Taskcluster artifacts
+
+On a tag push, we trigger another task:
+
+- `github_release` will create a release with notes on Github
+
+On a deployment (push on `testing` or `production`), we trigger those deployment tasks:
+
+- `backend_deploy` deploys the backend docker image on Heroku web dyno
+- `events_deploy` deploys the events docker image on Heroku web & worker dynos
+- `integration_deploy` pushes the integration docker image on a docker registry
+- `integration_hook` updates the integration Taskcluster hook to use the new docker image

docs/ci-cd/community.mermaid

@@ -0,0 +1,36 @@
+graph TD
+
+push --> check_lint
+push --> backend_check_tests
+push --> events_check_tests
+push --> frontend_build
+push --> integration_check_tests
+check_lint --> events_build
+events_check_tests --> events_build
+check_lint --> backend_build
+backend_check_tests --> backend_build
+check_lint --> integration_build
+integration_check_tests --> integration_build
+
+frontend_build --> frontend_deploy
+events_build --> events_deploy
+backend_build --> backend_deploy
+integration_build --> integration_deploy
+
+subgraph deployment
+  events_deploy
+  frontend_deploy
+  backend_deploy
+  integration_deploy
+  integration_deploy --> integration_hook
+end
+
+
+backend_build --> github_release
+events_build --> github_release
+frontend_build --> github_release
+integration_build --> github_release
+
+subgraph release
+  github_release
+end

docs/ci-cd/firefox-ci.mermaid

@@ -0,0 +1,6 @@
+graph TD
+
+push --> check_lint
+push --> bot_check_tests
+check_lint --> bot_build_dind
+bot_check_tests --> bot_build_dind

docs/configuration.md

@@ -0,0 +1,194 @@
+# Configuration
+
+The whole project uses [Taskcluster secrets](https://docs.taskcluster.net/docs/manual/using/secrets) as a source of configuration.
+We use a shared secret amongst every actor of the system, with a hierarchy so that common configurations can be easily shared without duplication.
+
+The source code to manage secrets is in the [Taskcluster python library](https://github.com/taskcluster/taskcluster/blob/master/clients/client-py/taskcluster/helper.py) itself.
+
+A developer can also specify its local configuration using a YAML file, with the same exact structure as the secret described below.
+
+We use one secret per environment, on the firefox-ci Taskcluster instance:
+
+- `project/relman/code-review/runtime-testing` for the testing environment
+- `project/relman/code-review/runtime-production` for the production environment
+
+## Sample commented file
+
+```yaml
+# Configuration under the `common` key is shared across the 3 other Python projects (bot, backend & events)
+common:
+
+  # This defines the environment name, used to customize logging, reporting, ...
+  # Usual choices are: `testing`, `production` or `localdev`, `dev`
+  APP_CHANNEL: testing
+
+  # This section defines the optional statistics connection for an InfluxDB database.
+  # This system is used and deployed by Mozilla Cloudops team.
+  # Of course, it must be disabled on local configurations.
+  influxdb:
+    ssl: true
+    password: xxxx
+    username: yyyy
+    database: zzzz
+    port: 8086
+    host: something.influxcloud.net
+
+  # These two keys allow the project to log everything on Papertrail, so that administrators can troubleshoot the system.
+  # For more information, see the debugging documentation.
+  # Of course, it must be disabled on local configurations.
+  PAPERTRAIL_HOST: logs.papertrailapp.com
+  PAPERTRAIL_PORT: 12345
+
+  # A common Phabricator account is shared across the bot and events projects
+  # Issues will be published with this account
+  # This section is required in local configuration
+  PHABRICATOR:
+    url: 'https://phabricator.services.mozilla.com/api/'
+    api_key: api-xxxx
+    publish: true
+
+  repositories:
+      # A unique display name for the repository
+    - name: mozilla-central
+
+      # The repository base url, used to clone the repository
+      # and reference it in the backend
+      url: 'https://hg.mozilla.org/mozilla-central'
+
+      # Try server configuration
+      # try_url is the ssh connection string to push patches
+      try_url: 'ssh://hg.mozilla.org/try'
+
+      # try_name is a display name for that repository
+      try_name: try
+
+      # try_mode describes how the try_task_config is published
+      # - json, to build a json v2 try_task_config.json
+      # - syntax, to add a commit with a try syntax as commit message
+      try_mode: json
+
+      # try_syntax is the full commit message used when try_mode = syntax
+      try_syntax: -p xxx -l code-review
+
+      # Mercurial checkout mode, several modes available:
+      # - robust, to use robustcheckout extension (recommended)
+      # - batch, to clone from revision 1 up to tip (slow but lower memory usage)
+      # - default, to use the default hg clone (for small repositories only)
+      checkout: robust
+
+      # Environment variables used by the bot to detect which repository
+      # is setup from a decision task (more details on the bot documentation)
+      decision_env_revision: GECKO_HEAD_REV
+      decision_env_repository: GECKO_HEAD_REPOSITORY
+
+      # The ssh username (or email) used to push on Try
+      ssh_user: someone@mozilla.com
+
+      # (Optional) private ssh-key overriding the default one setup in the events main section
+      ssh_key: xxxx
+
+# This main section is only used by the bot project
+bot:
+  # The list of reporter classes used to publish issues
+  REPORTERS:
+
+    # The most important one being Phabricator publication
+    # You just need to add that line to use the shared phabricator credentials
+    # Every issues deemed publishable will then be posted on the revision
+    - reporter: phabricator
+
+    # A debugging tool to receive an email on *every* build processed
+    # with the full list of issues
+    # Internal use only !
+    - reporter: mail
+      emails:
+        - admin@mozilla.com
+
+    # This reporter will send an extra email to the developers when a build error
+    # is detected amongst the issues
+    - reporter: build_error
+
+  # Boolean option to enable/disable the low coverage warning
+  ZERO_COVERAGE_ENABLED: true
+
+  # Connection information to publish issues on the backend
+  # On local development it should be set to the local backend running in Docker
+  backend:
+    url: 'https://api.code-review.moz.tools'
+
+    # That user can be created through the administration interface
+    username: xxxx
+    password: yyy
+
+  # Sentry connection string to publish system exceptions
+  # Each project has its own Sentry environment
+  SENTRY_DSN: https://xxx:yyy@sentry.com
+
+# This main section is only used by the backend project
+# Note: The postgresql connection string is provided by Heroku as an environment variable
+backend:
+
+  # Configure a list of allowed domains through CORS
+  # Useful for another frontend to use data from that backend
+  # Like on the Taskcluster builds
+  cors-domains:
+    - 'https://community.taskcluster-artifacts.net'
+
+  # Sentry connection string to publish system exceptions
+  # Each project has its own Sentry environment
+  SENTRY_DSN: https://xxx:yyy@sentry.com
+
+
+# This main section is only used by the events project
+# Note: The redis connection string is provided by Heroku as an environment variable
+events:
+  # Sentry connection string to publish system exceptions
+  # Each project has its own Sentry environment
+  SENTRY_DSN: https://xxx:yyy@sentry.com
+
+  # Does the system need to listen for autoland payloads on pulse
+  # This is needed to ingest autoland issues in the backend
+  autoland_enabled: true
+
+  # Pulse authentication to get messages for the autoland triggers
+  pulse_user: xxx
+  pulse_password: yyy
+
+  # Parameters for the test selection feature
+  # TODO: Marco may document a bit more here
+  bugbug_phabricator_deployment: dev
+  test_selection_enabled: false
+  test_selection_share: 0.03
+  test_selection_notify_addresses:
+    - admin@mozilla.com
+
+  # Skip all revisions produced by these users
+  # It's useful to avoid huge revisions produced by bots
+  user_blacklist:
+    - bot-username-a
+    - bot-username-b
+
+  # Taskcluster credentials for the community instance
+  # Needed to run the bugbug analysis
+  taskcluster_community:
+    client_id: project/relman/bugbug/xxxx
+    access_token: yyy
+
+  # Pulse authentication on the community Taskcluster instance
+  # to get messages for the test selection triggers
+  communitytc_pulse_user: xxxx
+  communitytc_pulse_password: yyy
+
+  # Phabricator usernames for whom a risk analysis will be triggered
+  # when they are authors or reviewers of a patch
+  risk_analysis_users:
+    - user-a
+    - user-b
+
+  # The SSH private key used to push patches on the Try server
+  # It can be setup globally for the events project, or specified per repository
+  ssh_key: |-
+    -----BEGIN RSA PRIVATE KEY-----
+    SomePrivateKeyHere
+    -----END RSA PRIVATE KEY-----
+```

docs/debugging.md

@@ -0,0 +1,113 @@
+# Debugging the code review bot
+
+As a maintainer on the code review bot, you will face errors or bad behaviours. There are a lot of moving parts in the projects, some of them are not directly under our control...
+
+## Tools for live environments
+
+### Live logs with papertrail
+
+The 3 *back-end* projects (bot, backend & events) all use [papertrail](https://my.papertrailapp.com) to log every Python `logging` call.
+
+The events are organized with a hierarchy so that you can filter:
+
+- `program:code-review` will display all code review logs from all environments
+- `program:code-review/production` will display all code review logs from production
+- `program:code-review/production/events` will display all code review logs from events in production
+
+To get access to Papertrail, you need to file a bug and ask `:catlee` for permissions to access the `sallt-team` group (see [Bug 1580134](https://bugzilla.mozilla.org/show_bug.cgi?id=1580134)).
+
+### Exceptions with Sentry
+
+Every exception for the 3 *back-end* projects (bot, backend & events) are logged on the Mozilla sentry instance.
+
+To get access, you need to file a bug requesting access to the `sallt` project on https://sentry.prod.mozaws.net/. The bug should be assigned to cloudops.
+
+Each project has its own dedicated space on Sentry, allowing you to filter and search issues.
+
+You can also configure Sentry to notify you by email on new exceptions.
+
+### Statistics
+
+Mozilla can use the official influxdb platform to process statistics. To get access to the platform, you need to file a bug, requesting access to cloudops.
+
+You'll then get access to the [code review dashboard](https://earthangel-b40313e5.influxcloud.net/d/Tat_-20Wz/code-review?orgId=1) showing the last 2 weeks of statistics from the code review bot.
+
+It's possible to configure alerts on your account when a metric goes under a certain level.
+
+### Heroku
+
+You'll need access to the `mozillacorporation` group on Heroku. Request on Bugzilla to be added in the mozillian group `heroku-members`. You'll then be able to login through SSO on Heroku (detailed and up to date information on [wiki.mozilla.org](https://wiki.mozilla.org/ReleaseEngineering/How_To/Heroku_CLI)).
+Once on Heroku, an existing admin for the code-review platform needs to upgrade your rights so you can manage the related applications.
+
+Finally you'll be able to see recent logs for the code review dynos (backend & events). That's helpful if Papertrail has issues.
+
+### Frontend
+
+The code review frontend hosted on https://code-review.moz.tools (production) and https://code-review.testing.moz.tools (testing) offers a pretty good overview of what is going on in the system.
+
+If you do not have a bunch of reviews in a normal week day on the front page under 20 minutes, something is probably going haywire.
+
+The old version using Taskcluster raw data is still available under https://code-review.moz.tools/#/tasks and will list (slowly) all the publication tasks with their ending tasks. Seeing a lot of red is never a good sign.
+
+### Integration test
+
+The repository also hosts an **integration test**. It's a Python script running daily as a Taskcluster hook to check that the whole stack is working.
+
+You can update its hook in Taskcluster, to send you an email once a revision is applied on phabricator-dev.
+
+If the daily revision from the integration test does not show up, it's a good indicator that something is broken.
+
+
+### Human probes
+
+Also known as "developers will complain". Our customers are great, and some of them rely heavily on the code review bot.
+They will tell you if something goes wrong, as the code review bot adds a link to post issues on every comment, and by now they know where to reach the team on Matrix #code-review-bot.
+
+## Is the platform still running ?
+
+Here is a list of troubleshooting steps when you *know* that something does not work, but don't know yet which part is buggy:
+
+- Check the frontend. As mentioned above, that's the easiest and fastest way to see what's going on in real time
+- Check the logs. Start by looking for the events logs, as that's the first piece that could fail in the workflow.
+- Check events on Heroku: are the dynos running (both web & worker) ?
+- Is the redis database full ?
+- If events is behaving normally, applying revisions, pick a try job from the logs, and follow it
+- Check the decision task is creating analyzers and the code-review ending task
+- Check the analyzers create some json artifacts, and look for incoherent data in the output. Are the tasks in a coherent status?
+- Check the code review hook on firefox-ci: Is it triggered ? Are there bot jobs in papertrail ? Can you find the bot task for the try task you were looking before in the logs ?
+- Check the bot execution on the job, through its Taskcluster logs. Check the build updates on Phabricator.
+- Check the backend is running, through Papertrail or Heroku logs. Is the postgresql database full ?
+
+## Testing changes
+
+When developing a new feature in the stack, it's primordial to test changes before shipping anything. You especially want to check the output on Phabricator for a known set of issues.
+
+### Running locally on same revision
+
+As a developer on the platform, you must be able to check your code changes locally. Each project has its own way to run, but the bot is the most interesting part as that's the most *public facing* part.
+
+A simple test is to pick a known try job with a set of issues (you need to know the task group id, and the code-review issues task id). Then you can run the bot before and after your changes with the Phabricator reporter disabled so you do not pollute an existing revision.
+
+```bash
+export TRY_TASK_GROUP_ID="xxx"
+export TRY_TASK_ID="yyy"
+code-review-bot -c code-review-local-dev.yml
+```
+
+This small boot script will use a local configuration `code-review-local-dev.yml` (that's where you disable the reporters) aiming at a given task group with issues.
+
+### Using phabricator-dev
+
+You could also configure the setup above to publish on Phabricator dev : https://phabricator-dev.allizom.org
+
+If you create a dummy revision on that instance (creating a bad patch, and publishing it with moz-phab on that Phabricator instance), you will get your own treeherder job, and more importantly your own Phabricator revision on a separate server.
+
+Here is a good example: https://phabricator-dev.allizom.org/D1758
+
+The testing environment is normally configured to publish issues on that Phabricator instance.
+
+### Shadow mode
+
+It's also possible to run the code review testing instance in **shadow mode**.
+
+In that setup, the bot project has phabricator credentials for the production instance, and the Phabricator reporter disabled. It will then be triggered for every patch published in production (so a lot more than on phabricator-dev), without publishing results (or you would pollute every revision with dupes !). You can then analyze the behaviour of the bot through its logs, and its crash rate, using a real flow of patches.

docs/phabricator.md

@@ -0,0 +1,81 @@
+# Phabricator
+
+The main goal of the project is to publish issues on Phabricator, so we need a good integration with their API and interface.
+
+## Structure
+
+![](phabricator.png)
+[Graph source](phabricator.mermaid)
+
+Developers interact with Phabricator **Revision** objects: when they push a patch, it creates a new revision.
+
+The patch is a **Diff**, linked to a unique **Revision**. When a developer updates its patch, the **Revision** gets a new **Diff**. The latest **Diff** is displayed by default when viewing a **Revision** on the Phabricator web interface.
+
+Phabricator instances can be configured with [Herald rules](https://secure.phabricator.com/book/phabricator/article/herald/) to react to internal events. This allows us to get a new [Harbormaster build](https://secure.phabricator.com/book/phabricator/article/harbormaster/) for every new patch.
+
+An **Harbormaster build** is made so that external tools (like the code review bot) can analyze a **Diff** and report back on its status. It can have 3 states:
+
+- `work` when the build is in progress,
+- `pass` when the build is valid (no issues),
+- `fail` when an issue was detected.
+
+The `pass` and `fail` states are final. Once reported on the build, its state cannot be updated anymore.
+
+It's possible to restart a build (as an administrator on production, as owner of the revision in testing), so you can have multiple builds per **Diff**.
+
+When some issues are detected, they are reported as **LintResult**. Phabricator will display them as a list at the top of the **Revision**, with their name, and code, linking to their location in the patch.
+
+Unit test failures could also be reported through **UnitResult** in the same way.
+
+When a build is restarted, or a new diff is created, previous results are cleared and not shown anymore.
+
+Lastly it's possible to publish a **Comment** on the revision, but it will not be tied to a **Diff** or **Harbormaster Build**. We use that to publish generic and persistent information about issues (the summary comment).
+
+
+## API
+
+To interact with Phabricator, we need to use their *~REST* API. It's self documented on the `/conduit` endpoint:
+
+- https://phabricator.services.mozilla.com/conduit/ for the production instance
+- https://phabricator-dev.allizom.org/conduit/ for the testing instance
+
+Our Phabricator API client is developed in a separate repository [libmozdata](https://github.com/mozilla/libmozdata/blob/master/libmozdata/phabricator.py), but still maintained by the same team.
+
+Please note that the Phabricator API use a *weird* style to pass parameters (only POST requests, with a JSON payload inside a dict urlencoded: use the existing client instead of writing your own).
+
+## Endpoints
+
+All the endpoints we use in the project are implemented with some high level function in the Phabricator API client hosted in libmozdata.
+
+| Endpoint                     | Description                                                                                 |
+|------------------------------|---------------------------------------------------------------------------------------------|
+| differential.createcomment   | Publish a comment on a Revision                                                             |
+| differential.diff.search     | Lookup a Phabricator Diff                                                                   |
+| differential.getrawdiff      | Get the raw patch for a given Diff, hosted on Phabricator, not HGMO                         |
+| differential.revision.search | Lookup a Phabricator Revision                                                               |
+| diffusion.repository.search  | Get details for a repository, or a list of supported repositories                           |
+| harbormaster.build.search    | Search for a build on a diff/revision                                                       |
+| harbormaster.createartifact  | An artifact can be a file or a link: used to publish the treeherder link on a Build.        |
+| harbormaster.sendmessage     | Update a build state, and publishes results (issues).                                       |
+| transaction.search           | List internal transactions for an object, to identify if a comment has already been posted. |
+| user.whoami                  | Load current authenticated user, to check credentials.                                      |
+
+## Subtleties
+
+### ID and PHID
+
+Phabricator uses two IDs per object: a numerical value `ID`, which is exposed to users, and an internal ID, called the `PHID`, largely used in the API (it looks like `PHID-XXX-YYYYYY`)
+
+### Search
+
+To get details about a specific object, you need to use the `xxx.search` endpoint with an ID constraint. There is no `xxx.details` endpoint. You'll see that a lot in libmozdata.
+
+
+### Unit tests
+
+There is an extensive test suite in the bot, using several mockups to build fake Phabricator [responses](https://github.com/getsentry/responses). The Conduit team has also a great test system in the lando-api project - it would be great to share that as a library.
+
+
+## Help
+
+The EngFlow and Conduit team will help you for troubleshooting issues and building great features: they're available on Matrix #conduit.

docs/phabricator.mermaid

@@ -0,0 +1,6 @@
+graph TD
+	Revision --> Diff
+  Revision --> Comment
+  Diff --> build[Harbormaster Build]
+  build --> LintResult
+  build --> UnitResult

docs/projects/backend.md

@@ -0,0 +1,60 @@
+# Code review Backend
+
+This is a Python REST API, powered by [Django](https://docs.djangoproject.com/) and [django-rest-framework](https://www.django-rest-framework.org/).
+
+The technical information on how to get started is available in the [project README](/backend/README.md).
+
+This project is used by the bot and the frontend: the bot publishes issues at the end of each execution, and the frontend fetches data to display it nicely to users (admins or developers).
+
+## Deployment
+
+The application is hosted on Heroku (more information in [debugging](/docs/debugging.md) to get access).
+
+It uses currently a single web dyno on each environment:
+- https://api.code-review.moz.tools on production
+- https://api.code-review.testing.moz.tools on testing
+
+It is deployed with [task-boot](https://github.com/mozilla/task-boot) on every push to `testing` or `production` branch by an administrator.
+The application has no state in its docker image, it can restart immediately and reuse the same database.
+
+### Database migration
+
+If you want to make a Database migration (adding or altering a model), you'll need to make a new deployment, then run the following command from your computer using the [Heroku cli](https://devcenter.heroku.com/articles/heroku-cli).
+
+Make sure you're authenticated first (it will use your browser to get your SSO credentials):
+
+```bash
+heroku login
+```
+
+Then run the migration on the desired application:
+
+```bash
+heroku run -a code-review-backend-production ./manage.py migrate
+```
+
+It's also possible to do that through the web shell on the Heroku dashboard.
+
+### Administration
+
+You can also use the heroku cli application to add a new administrator account:
+
+```bash
+heroku run -a code-review-backend-production ./manage.py createsuperuser
+```
+
+The admin dashboard is available as `/admin/` on each instance, login with the credentials from the above command.
+
+## Models
+
+![](backend.png)
+
+Generated with [django-extensions](https://django-extensions.readthedocs.io/en/latest/graph_models.html)
+
+TODO: explain a bit Issue models
+
+## Endpoints
+
+All endpoints are described in the generated [OpenAPI documentation](https://api.code-review.moz.tools/docs).
+
+The OpenAPI raw file is available as [JSON](https://api.code-review.moz.tools/docs.json) and [YAML](https://api.code-review.moz.tools/docs.yaml)

docs/projects/bot.md

@@ -0,0 +1,111 @@
+# Code review Bot
+
+The publication task (also called **bot**) is a Python script, running as a Taskcluster task.
+
+The technical information on how to get started is available in the [project README](/bot/README.md).
+
+## Deployment
+
+The project is built as a Docker in Docker image, on the firefox-ci instance (more information in [CI/CD](/docs/ci-cd/README.md)).
+
+Two hooks are available and configured through [ci-configuration](https://hg.mozilla.org/ci/ci-configuration/file/tip/hooks.yml):
+
+- [project-relman/code-review-production](https://firefox-ci-tc.services.mozilla.com/hooks/project-relman/code-review-production)
+- [project-relman/code-review-testing](https://firefox-ci-tc.services.mozilla.com/hooks/project-relman/code-review-testing)
+
+## Workflow
+
+The overall workflow of the publication task is relatively simple:
+
+- The code review hook is triggered by a pulse message, starting a new task for each completed analysis
+- List all Taskcluster tasks in the analysis
+- Instantiate a task parser for each taskcluster task
+- Build issues for each tasks
+- Aggregate them
+- Publish them on Phabricator
+
+![](bot.png)
+[Graph source](bot.mermaid)
+
+### Publishable issues
+
+We publish only a subset of issues detected:
+
+- every **Error** is reported, no matter where it's detected.
+- **Warnings** that are inside the patch (modified lines of files in patch) are reported
+
+Other warnings are discarded (but still reported in debug and backend mode)
+
+## Supported tasks
+
+Each analyzer must be supported by the bot, in order to convert the JSON artifacts into a list of issues.
+
+The first half part of the workflow is to convert JSON files into a list of `code_review_bot.Issue` sub classes instances.
+
+Each `code_review_bot.Issue` has a common interface to build Phabricator LintResult and comments (amongst other outputs).
+
+### Clang-Tidy
+
+This supports the `source-test-clang-tidy` task from mozilla-central, and parses the custom artifact `public/code-review/clang-tidy.json`.
+
+It will output a list of `ClangTidyIssue`, reporting static analysis warnings. A few extra filtering rules are present.
+
+### Clang-Format
+
+This supports the `source-test-clang-format` task from mozilla-central, and parses the custom artifact `public/code-review/clang-tidy.diff`. (Only task to use a diff directly)
+
+It will output a list of `ClangFormatIssue`, reporting formatting warnings, with fixes provided to the developer.
+
+### Coverity
+
+This supports the Coverity integration from mozilla-central and NSS, and parses the custom artifact `public/code-review/coverity.json`.
+
+It will output a list of `CoverityIssue`, reporting static analysis warnings and build errors.
+
+### Infer
+
+This supports the `source-test-infer` from mozilla-central, and parses the custom artifact `public/code-review/infer.json`.
+
+It will output a list of `InferIssue` reporting static analysis and formatting issues in Java source code.
+
+### Mozlint
+
+This supports all the Mozlint tasks (and there are a lot!) from mozilla-central. It parses the custom artifact and mozlint json format defined in `public/code-review/mozlint.json`.
+
+It will output a list of `MozLintIssue` reporting on various issues for most languages in Mozilla central.
+
+### Default format
+
+We also built a default task support & format, described in extensive details [here](/docs/analysis_format.md).
+
+This format should be used by new analyzers, as this does not require any change in the bot!
+
+## Reporters
+
+The publication task support several reporters to publish issues. They are defined in the shared [configuration](/docs/configuration.md).
+
+### Phabricator
+
+This is definitely the most important reporter. It will publish all issues deemed publishable on the Phabricator revision & build being analysed.
+
+We publish two things (when issues are published):
+- a summary comment listing the number of issues found, their type, and some help text
+- one **LintResult** per issue found.
+
+For more information, read the [phabricator](/docs/phabricator.md) documentation.
+
+### Debug
+
+The debug reporter lists all the issues in a public JSON artifact and sends an email to admins with the full list of issues.
+
+It could be deprecated now that we have better tracking and management of issues (it was the first reporter created, a long time ago).
+
+### Build error
+
+Build errors are really bad, and we want to send an email to the developer when they occur. So this reporter simply sends an email to the developer using Taskcluster notification system.
+
+### Backend
+
+The backend has no reporter as it's more tightly coupled to the bot system, but every issue is published on the backend.
+
+Each Issue also has a unique hash calculated, using the modified lines source code, the issue summary. It's used to be able to compare issues between each other.

docs/projects/bot.mermaid

@@ -0,0 +1,12 @@
+graph TD
+  pulse[Pulse Trigger] -- task id --> new
+  new[New bot task] --> tasks
+  tasks{List all tasks} -->|artifact| clang-tidy
+  tasks{List all tasks} -->|artifact| mozlint
+  tasks{List all tasks} -->|artifact| analyzer-X
+  clang-tidy --> issues[List of issues]
+  mozlint --> issues
+  analyzer-X --> issues
+  issues --> reporters{Publication}
+  reporters --> Phabricator
+  reporters --> Mail

docs/projects/events.md

@@ -0,0 +1,41 @@
+# Code review Events
+
+This is a Python distributed application, split between a public API endpoint and mercurial workers, all running in the same Heroku application.
+
+The technical information on how to get started is available in the [project README](/events/README.md).
+
+This project is used by the Phabricator Harbormaster build plan: every patch published sends an HTTP request on the public endpoint, which in turns get stored in a shared Redis database, then the build's patch is applied and pushed to Try.
+
+## Deployment
+
+The application is hosted on Heroku (more information in [debugging](/docs/debugging.md) to get access).
+
+It uses currently a single web dyno on each environment:
+- https://events.code-review.moz.tools on production
+- https://events.code-review.testing.moz.tools on testing
+
+It is deployed with [task-boot](https://github.com/mozilla/task-boot) on every push to `testing` or `production` branch by an administrator.
+The application has no state in its docker image, it can restart immediately and reuse the same database.
+
+The worker dynos are slow to start though, as they need to clone the repositories described in the [configuration](/docs/configuration.md). Current boot time is ~ 10 minutes.
+
+## Workflow
+
+Here is a sequence diagram showcasing the main exchanges between relevant parties:
+- Phabricator
+- the **web** dyno on Heroku
+- the shared Redis database on Heroku
+- one of the mercurial **workers** on Heroku
+- the Try server
+
+![](events.png)
+[Diagram source](events.mermaid)
+
+As you can see the workflow is quite complex, relying on several distributed parts. We decided to create an external library [libmozevent](https://github.com/mozilla/libmozevent) to abstract most of the exchanges.
+
+That library hosts all the code relevant to mercurial workers, Phabricator interactions, and uses a bus system to communicate between sub systems.
+This allows us to have the same code run in a single process in development, but across several distributed instances in testing & production (on Heroku), linked together through a Redis database.
+
+Each subsystem has input and output queues, and just consumes input queues to get new data to work on (whatever that data may be). Once its process is done, it puts in the relevant output queues the results (it's basically a plugin system, split on a network).
+
+A big advantage of that system is that the high level code in this repository is relatively simple, and mainly *plugs* the right subsystems together. It's also easier to create unit tests for each subsystem, as you can interact with their queues.

docs/projects/events.mermaid

@@ -0,0 +1,17 @@
+sequenceDiagram
+
+  Phabricator->>+Web: GET /codereview/new
+  Web->>+Redis: Store build reference X
+  Worker->>Redis: Got any build ?
+  Redis->>Worker: Here is build X
+  Worker->>+Phabricator: Is build X publicly available ?
+  Phabricator->>+Worker: No, it's still private
+  Worker->>Worker: Let's wait a bit
+  Worker->>+Phabricator: Is build X publicly available ?
+  Phabricator->>+Worker: Yes.
+  Worker->>+Phabricator: Give me the stack of X
+  Phabricator->>+Worker: Here is a list of patches
+  Worker->>Worker: Let's apply that stack on my local clone
+  Worker->>Try: hg push -f try
+  Try->>Worker: ok, it will be on treeherder as XYZ
+  Worker->>Phabricator: Here is the treeherder link XYZ for build X

docs/projects/frontend.md

@@ -0,0 +1,20 @@
+# Code Review Frontend
+
+The frontend is a Single Page Application, built with [Vue.JS](https://vuejs.org), using [vuex](https://vuex.vuejs.org/) as a store for data shared across components in the application.
+
+It's really a pretty simple application, not much complexity:
+- few views, and few components
+- uses [axios](https://github.com/axios/axios) to retrieve data from the backend
+- no authentication, everything is public
+- uses [chartist](https://gionkunz.github.io/chartist-js/) to build the stats graph
+- uses [vue-router](https://router.vuejs.org/) to handle routing
+
+The application is built with [neutrino](https://neutrinojs.org/) (a Mozilla project) using its defaults for Vue.js applications.
+
+On every Github push (pull request or branch), the frontend is built, and even usable from the Taskcluster artifacts (it uses the testing environment as its default source).
+
+The application is then deployed with [task-boot](https://github.com/mozilla/task-boot/) on an Amazon S3 bucket, exposed through a Cloudfront configuration (this is managed by the Cloudops team at Mozilla).
+
+Finally the application is currently available at:
+- https://code-review.moz.tools/ on production (uses production backend)
+- https://code-review.testing.moz.tools/ on testing (uses testing backend)

docs/publication.md

@@ -39,5 +39,5 @@ We have [plans](https://bugzilla.mozilla.org/show_bug.cgi?id=1555721) to remove
 2. Check that your task is present in the task group published by the code review bot as `Treeherder jobs`
 3. Check that your task produces the expected analysis artifact
 4. Check that the `code-review-issues` is present in that task group (for mozilla-central tasks)
-5. Check that your test diff is available on [our dashboard](https://static-analysis.moz.tools/) by searching its revision ID or title (it can take several seconds to load all the tasks available)
+5. Check that your test diff is available on [our dashboard](https://code-review.moz.tools/) by searching its revision ID or title (it can take several seconds to load all the tasks available)
 6. Reach out to us, see [README](../README.md) for contact info

frontend/README.md

@@ -1,7 +1,7 @@
 Code Review Frontend
 ====================
 
-This is a simple Vue.JS administration frontend, the production instance is publicly available at https://static-analysis.moz.tools/
+This is a simple Vue.JS administration frontend, the production instance is publicly available at https://code-review.moz.tools/
 
 Developer setup
 ---------------