From 030b2547dc12b01477d11871971782bac2608db1 Mon Sep 17 00:00:00 2001 From: Armen Zambrano Date: Wed, 27 May 2020 08:25:45 -0400 Subject: [PATCH] docs: Drop info about submitting data via Pulse (#6448) --- docs/common_tasks.md | 11 ---- docs/submitting_data.md | 141 +++------------------------------------- 2 files changed, 9 insertions(+), 143 deletions(-) diff --git a/docs/common_tasks.md b/docs/common_tasks.md index d658cfb96..617e542e1 100644 --- a/docs/common_tasks.md +++ b/docs/common_tasks.md @@ -26,17 +26,6 @@ The docs will then be available at: [yarn documentation]: https://yarnpkg.com/en/docs/usage -## Add a new Mercurial repository - -To add a new repository, the following steps are needed: - -- Append new repository information to the fixtures file located at: - `treeherder/model/fixtures/repository.json` -- Restart any running Django runserver/Celery processes. - -For more information on adding a new GitHub repository, see -[Adding a GitHub repository](submitting_data.md#adding-a-github-repository). - ## Debugging Tools You can use the Python Debugger ([pdb](https://docs.python.org/3.7/library/pdb.html)) in a Docker container. diff --git a/docs/submitting_data.md b/docs/submitting_data.md index de634484c..a6c731c44 100644 --- a/docs/submitting_data.md +++ b/docs/submitting_data.md @@ -1,136 +1,13 @@ # Submitting data to Treeherder -To submit your test data to Treeherder, you have two options: - -1. Using Taskcluster (recommended) - - It's strongly recommended that you schedule jobs using Taskcluster rather - than using your own build system, since otherwise many Treeherder features - will not work without significant effort, and it's also unlikely your jobs - will be eligible for tier 1 or even tier 2 status. - -2. [Using Pulse](#using-pulse) - - This is the process Taskcluster is using to submit data to Treeherder. - There is a [Pulse Job Schema] to validate your payload against to ensure it will - be accepted. In this case, you create your own [Pulse] exchange and publish - to it. To get Treeherder to receive your data, you would create a bug to - have your Exchange added to Treeherder's config. All Treeherder instances - can subscribe to get your data, as can local dev instances for testing. - - While it is beyond the scope of this document to explain how [Pulse] and - RabbitMQ work, we encourage you to read more about this technology on - its Wiki page. - - Note: Support for submitting data using Pulse outside of Taskcluster may - be removed in the future. +To submit your data to Treeherder you need your tasks running on Taskcluster. If you are establishing a new repository with Treeherder, then you will need to do one of the following: 1. For GitHub repos: [Adding a GitHub Repository](#adding-a-github-repository) -2. For Mercurial repos: [Add a new Mercurial repository](common_tasks.md#add-a-new-mercurial-repository) - -## Using Pulse - -To submit via a Pulse exchange, these are the steps you will need to follow: - -### 1. Format your data - -You should format your job data according to the [Pulse Job Schema], -which describes the various properties of a job: whether it passed or failed, -job group/type symbol, description, log information, etc. -You are responsible for validating your data prior to publishing it onto your -exchange, or Treeherder may reject it. - -### 2. Create your Exchange - -With [Pulse Guardian], you need to create your Pulse User in order to -create your own Queues and Exchanges. There is no mechanism to create an -Exchange in the Pulse Guardian UI itself, however. You will need to create -your exchange in your submitting code. There are a few options available -for that: - -1. [MozillaPulse] -2. [Kombu] -3. Or any RabbitMQ package of your choice - -To test publishing your data to your Exchange, you can use the Treeherder -management command [publish_to_pulse]. This is also a very simple example -of a Pulse publisher using Kombu that you can use to learn to write your own -publisher. - -### 3. Register with Treeherder - -Once you have successfully tested a round-trip through your Pulse exchange to -your development instance, you are ready to have Treeherder receive your data. - -Treeherder has to know about your exchange and which routing keys to use in -order to load your jobs. - -Submit a [Treeherder bug] with the following information: - -```python -{ - "exchange": "exchange/my-pulse-user/v1/jobs", - "projects": [ - 'mozilla-inbound._' - ], -}, -``` - -Treeherder will bind to the exchange looking for all combinations of it and the -`projects`. For example with the above config, we will only load jobs from -the `mozilla-inbound._` project. - -If you want all jobs from your exchange to be loaded, you can use the `#` -wildcard like so: - -```python -"projects": [ - '#' -], -``` - -If you want one config to go to Treeherder Staging and a different one to go -to Production, please specify that in the bug. You could use the same exchange -with different project settings, or two separate exchanges. The choice is -yours. - -### 4. Publish jobs to your Exchange - -Once the above config is set on Treeherder, you can begin publishing jobs -to your Exchange and they will start showing in Treeherder. - -You will no longer need any special credentials. You publish messages to the -Exchange YOU own. Treeherder is now just listening to it. - -## Schema Validation - -Some data types in Treeherder will have JSON Schema files in the form of YAML. -You can use these files to validate your data prior to submission to be sure -it is in the right format. - -You can find all our data schemas in the [schemas] folder. - -To validate your file against a `yml` file, you can use something like the -following example code: - -```python -import yaml -import jsonschema - -with open('schemas/text-log-summary-artifact.yml') as f: - schema = yaml.load(f, Loader=yaml.FullLoader) - -jsonschema.validate(data, schema) -``` - -This will give output telling you if your `data` element passes validation, -and, if not, exactly where it is out of compliance. - -[schemas]: https://github.com/mozilla/treeherder/tree/master/schemas +2. For Mercurial repos: [Add a new Mercurial repository](#adding-a-mercurial-repository) ## Adding a GitHub Repository @@ -151,10 +28,10 @@ The following steps are required: - Answer: Are any of those repositories private? - State that this is only to get Pulse messages for integration into Treeherder -[pulse guardian]: https://pulseguardian.mozilla.org/whats_pulse -[pulse]: https://wiki.mozilla.org/Auto-tools/Projects/Pulse -[pulse job schema]: https://github.com/mozilla/treeherder/blob/master/schemas/pulse-job.yml -[treeherder bug]: https://bugzilla.mozilla.org/enter_bug.cgi?component=Treeherder:%20Data%20Ingestion&form_name=enter_bug&product=Tree%20Management -[mozillapulse]: https://pypi.python.org/pypi/MozillaPulse -[kombu]: https://pypi.python.org/pypi/kombu -[publish_to_pulse]: https://github.com/mozilla/treeherder/blob/master/treeherder/etl/management/commands/publish_to_pulse.py#L12-L12 +## Adding a Mercurial repository + +To add a new repository, the following steps are needed: + +- Append new repository information to the fixtures file located at: + `treeherder/model/fixtures/repository.json` +- Restart any running Django runserver/Celery processes.