From 24aaeb846156ac562e74f0d120cfb780d3760b7f Mon Sep 17 00:00:00 2001
From: Gregory Mierzwinski <gmierz2@outlook.com>
Date: Tue, 10 Dec 2019 15:29:06 +0000
Subject: [PATCH] Bug 1587080 - Part 2: Integrate perfdocs into mozlint. r=ahal

This patch integrates the performance documentation (perfdocs) verification into mozlint. It can be used with `./mach lint -l perfdocs`.

Depends on D53647

Differential Revision: https://phabricator.services.mozilla.com/D53648

--HG--
extra : moz-landing-system : lando
---
 tools/lint/docs/linters/perfdocs.rst | 84 ++++++++++++++++++++++++++++
 tools/lint/perfdocs.yml              | 11 ++++
 2 files changed, 95 insertions(+)
 create mode 100644 tools/lint/docs/linters/perfdocs.rst
 create mode 100644 tools/lint/perfdocs.yml

diff --git a/tools/lint/docs/linters/perfdocs.rst b/tools/lint/docs/linters/perfdocs.rst
new file mode 100644
index 000000000000..f7e548139e00
--- /dev/null
+++ b/tools/lint/docs/linters/perfdocs.rst
@@ -0,0 +1,84 @@
+PerfDocs
+========
+
+`PerfDocs`_ is a tool that checks to make sure all performance tests are documented in tree.
+
+At the moment, it is only used for this documentation verification, but in the future it will also auto-generate documentation from these descriptions that will be displayed in the source-docs documentation page (rather than the wiki, which is where they currently reside).
+
+Run Locally
+-----------
+
+The mozlint integration of PerfDocs can be run using mach:
+
+.. parsed-literal::
+
+    $ mach lint --linter perfdocs
+
+
+Configuration
+-------------
+
+There are no configuration options available for this linter. It scans the full source tree under ``testing``, looking for folders named ``perfdocs`` and then validates their content. This has only been implemented for Raptor so far, but Talos will be added in the future. We also hope to expand this to search outside the ``testing`` directory.
+
+The ``perfdocs`` folders, there needs to be an ``index.rst`` file and it needs to contain the string ``{documentation}`` in some location in the file which is where the test documentation will be placed. The folders must also have a ``config.yml`` file following this schema:
+
+.. code-block:: python
+
+    CONFIG_SCHEMA = {
+        "type": "object",
+        "properties": {
+            "name": {"type": "string"},
+            "manifest": {"type": "string"},
+            "suites": {
+                "type": "object",
+                "properties": {
+                    "suite_name": {
+                        "type": "object",
+                        "properties": {
+                            "tests": {
+                                "type": "object",
+                                "properties": {
+                                    "test_name": {"type": "string"},
+                                }
+                            },
+                            "description": {"type": "string"},
+                        },
+                        "required": [
+                            "description"
+                        ]
+                    }
+                }
+            }
+        },
+        "required": [
+            "name",
+            "manifest",
+            "suites"
+        ]
+    }
+
+Here is an example of a configuration file for the Raptor framework:
+
+.. parsed-literal::
+
+    name: raptor
+    manifest: testing/raptor/raptor/raptor.ini
+    suites:
+        desktop:
+            description: "Desktop tests."
+            tests:
+                raptor-tp6: "Raptor TP6 tests."
+        mobile:
+            description: "Mobile tests"
+        benchmarks:
+            description: "Benchmark tests."
+            tests:
+                wasm: "All wasm tests."
+
+Note that there needs to be a FrameworkGatherer implemented for the framework being documented since each of them may have different ways of parsing test manifests for the tests. See `RaptorGatherer <https://searchfox.org/mozilla-central/source/tools/lint/perfdocs/framework_gatherers.py>`_ for an example gatherer that was implemented for Raptor.
+
+Sources
+-------
+
+* `Configuration <https://searchfox.org/mozilla-central/source/tools/lint/perfdocs.yml>`_
+* `Source <https://searchfox.org/mozilla-central/source/tools/lint/perfdocs>`_
diff --git a/tools/lint/perfdocs.yml b/tools/lint/perfdocs.yml
new file mode 100644
index 000000000000..b81d5edf1417
--- /dev/null
+++ b/tools/lint/perfdocs.yml
@@ -0,0 +1,11 @@
+---
+perfdocs:
+    description: Performance Documentation linter
+    # This task handles its own search, so just include cwd
+    include: ['testing/raptor']
+    exclude: []
+    extensions: ['rst', 'ini', 'yml']
+    support-files: []
+    type: structured_log
+    payload: perfdocs:lint
+    setup: perfdocs:setup