Add pipeline documentation (#36522)

Co-authored-by: Grace Park <gracepark@github.com>

.github/workflows/sync-openapi.yml

@@ -112,4 +112,4 @@ jobs:
           channel: ${{ secrets.DOCS_ALERTS_SLACK_CHANNEL_ID }}
           bot-token: ${{ secrets.SLACK_DOCS_BOT_TOKEN }}
           color: failure
-          text: The last Sync OpenAPI schema run for ${{github.repository}} failed. See https://github.com/${{github.repository}}/actions/workflows/openapi-decorate.yml
+          text: The last Sync OpenAPI schema run for ${{github.repository}} failed. See https://github.com/${{github.repository}}/actions/workflows/sync-openapi.yml

src/automated-pipelines/README.md

@@ -1,11 +1,51 @@
 # Automated pipelines
 
-Our automated pipelines directory contains code shared by our automated pipelines, including REST, GraphQL, Webhooks, CodeQL, and GitHub Apps.
+Our automated pipelines directory contains code shared by our automated pipelines, including REST, GraphQL, Webhooks, CodeQL CLI, and GitHub Apps.
 
 ## What is automated pipelines
 
-An automated pipeline adds automatically generated content to docs.github.com
+An automated pipeline consumes data from an external source that is used to create content for docs.github.com. An automated pipeline does not automate documentation that is created by our content writing team. For example, if a writer creates a structured data file like YAML or JSON that lives in the `docs-internal` repo, using that data to create a page does not create an automated pipeline.
+
+Automated pages allow for manually created content to be prepended to the automated content, but do not allow for manually created content to be appended or interspersed within automated content. Manually created content (that is prepended to automated content) lives in the Markdown file associated with the automated page, along with the article's frontmatter metadata.
+
+## What automation pipelines are available
+
+- [CodeQL CLI](../codeql-cli/README.md)
+- [GitHub Apps](../github-apps/README.md)
+- [GraphQL](../graphql/README.md)
+- [REST](../rest/README.md)
+- [Webhooks](../webhooks/README.md)
 
 ## How does it work
 
+We currently have two patterns that we used to create automated pipelines:
+- REST, Webhooks, GitHub Apps, and GraphQL pipelines consume external structured data and transform that data into a JSON file that is used to create content for a page on docs.github.com. Typically, data files are a 1:1 mapping to a specific page on docs.github.com.
+- The CodeQL CLI pipeline takes an unstructured ReStructuredText file and transforms it directly into a Markdown file with frontmatter, that uses the same authoring format as the rest of the docs.
+
+## Creating a new pipeline
+
+Each pipeline should be evaluated individually to determine the best architecture for simplicity, maintainability, and requirements. 
+For example:
+- Is the content being displayed basic Markdown content? For example, does the content avoid using complex tables and interactive elements? If so, then writing the Markdown content directly and avoiding the need to create a structured data file that requires a React component may be the best approach. This was the case for the CodeQL CLI pipeline. One caveat to think about before writing Markdown directly is whether the content will need liquid versioning. The current pipeline that writes Markdown directly does not need to use liquid versioning. Liquid versioning which would increase the complexity quite a bit. All of the Markdown content in each article that is generated from the CodeQL CLI pipeline applies to all versions listed in the `versions` frontmatter property, simplifying the Markdown generation process.
+- Is the page interactive like the REST and Webhooks pages? If so, then the data will likely need to be structured data. In that case, a new React component may be needed to display the data.
+
+### Initial migrations
+
+When creating a new pipeline, the source data that is being consumed may not have all of the necessary data needed to create the page. Oftentimes, source data does not contain descriptions and prose that our content writers have crafted to describe properties or concepts. In this case, it's common to need to scrape data from our docs and merge it into a new field in the structured data file that we intend to consume. When creating a new pipeline, you'll need to work with the team that owns the source data to create a plan for adding any additional properties and agreeing on a format that will work best for both teams.
+
+### What to include in a new pipeline
+
+- Create a new directory in the `src` directory with the name of the pipeline. For example, `src/codeql-cli`.
+- Add a README.md file that describes the pipeline and how to use it. This should include any dependencies, how to run the pipeline, and any other information that is needed to use the pipeline. It's strongly recommended to include a diagram showing the overall flow of the pipeline.
+- Each pipeline typically requires a workflow to allow scheduling or manually running the pipeline. The workflow should be placed in the `.github/workflows` directory and named `sync-<pipeline-name>.js`. Each workflow typically requires adding a manual run option and an input parameter to specify the source repo's branch to use.
+- Each pipeline will need a `scripts` directory with (at minimum) a `scripts/sync.js` file to run the pipeline.
+- If the pipeline will contain structured data, you will need to add a `src/<pipeline-name>/data` directory. The files inside the `data` directory are typically organized by version (e.g., `src/webhooks/data/fpt/*`).
+- Pipelines typically have tests specific to the pipeline that are placed in the `src/<pipeline-name>/tests` directory. There is no need to add tests that render the page because all autogenerated pages are tested in `src/automated-pipelines/tests/rendering.js`.
+  - If the pipeline uses a Next.js page component (e.g., `pages/**/*.tsx`), ensure there is a test that fails if that page component is moved or deleted.
+
 ## How to get help
+
+Slack: `#docs-engineering`
+Repo: `github/docs-engineering`
+
+If you have a question about automation pipelines, you can ask in the `#docs-engineering` Slack channel. If you notice a problem with one of the automation pipelines, you can open an issue in the `github/docs-engineering` repository.

src/codeql-cli/README.md

@@ -1,11 +1,48 @@
-# CodeQL CLI
+# CodeQL CLI pipeline
 
-Our CodeQL CLI pipeline adds automated documentation on docs.github.com.
+Our CodeQL CLI pipeline creates autogenerated documentation for docs.github.com from the source code used to build the [CodeQL CLI](https://docs.github.com/en/code-security/codeql-cli/using-the-codeql-cli/about-the-codeql-cli).
 
-## What is CodeQL CLI
-
-CodeQL is an intelligent code scanning service that checks for security issues in code.
+The pipeline is used to generate Markdown files that create article pages on the docs.github.com site.
 
 ## How does it work
 
+![A flow chart describing how the automation pipeline for CodeQL CLI generates documentation](./codeql-cli-pipeline-flowchart.png)
+
+A [workflow](.github/workflows/sync-codeql-cli.yml) is used to trigger the automation of the CodeQL CLI documentation. The workflow is manually triggered by a member of the GitHub Docs team approximately every two weeks to align to releases of the CodeQL CLI. The workflow takes an input parameter that specifies the branch to pull the source files from in the semmle-code repo. If the branch input is omitted, the workflow will default to the `main` branch.
+
+The workflow runs the `src/codeql-cli/scripts/sync.js` script, which generates Markdown files under `content/code-security/codeql-cli/codeql-cli-manual`.
+
+The workflow automatically creates a new pull request with the changes and the label `codeql-cli-pipeline`.
+
+## Local development
+
+To run the CodeQL CLI pipeline locally:
+
+1. Clone the `semmle-code` repository inside your local `docs-internal` repository.
+2. [Install Pandoc](https://pandoc.org/installing.html). You can `brew install pandoc` on macOS.
+3. Run `src/codeql-cli/scripts/sync.js`.
+
+## About this directory
+
+- `src/rest/lib/config.json` - A configuration file used to specify metadata about the REST pipeline.
+- `src/rest/scripts` - The scripts and source code used run the CodeQL CLI pipeline. 
+  - `src/rest/scripts/sync.js` - The entrypoint script that runs the CodeQL CLI pipeline.
+
+## Content team
+
+The content writers can manually update parts of the autogenerated Markdown files in `content/code-security/codeql-cli/codeql-cli-manual`. When new Markdown files are added they will get all of the frontmatter properties defined in the `defaultFrontmatter` property in `src/codeql-cli/lib/config.js`. 
+
+When a new Markdown file is created, a writer can manually change any of the frontmatter. The pipeline will not overwrite the frontmatter on subsequent runs. 
+
+Writers can also add an introduction paragraph _above_ the following Markdown comment:
+
+```markdown
+<!-- Content after this section is automatically generated -->
+```
+
 ## How to get help
+
+Slack: `#docs-engineering`
+Repo: `github/docs-engineering`
+
+If you have a question about the CodeQL CLI pipeline, you can ask in the `#docs-engineering` Slack channel. If you notice a problem with the CodeQL CLI pipeline, you can open an issue in the `github/docs-engineering` repository.

src/github-apps/README.md

@@ -1,11 +1,56 @@
-# GitHub Apps
+# GitHub Apps pipeline
 
-Our GitHub Apps pipeline adds automated documentation for GitHub Apps.
+Our GitHub Apps pipeline creates autogenerated documentation for docs.github.com from the OpenAPI stored in the open-source repository [`github/rest-api-description`](https://github.com/github/rest-api-description).
 
-## What is GitHub Apps
-
-GitHub Apps is a GitHub product that allows custom applications to respond to events on GitHub.
+The pipeline is used to generate data that is used by the docs.github.com site when deployed locally, in preview environments, or in production.
 
 ## How does it work
 
+![A flow chart describing how the automation pipeline for GitHub Apps generates documentation](./github-apps-pipeline-flowchart.png)
+
+A [workflow](.github/workflows/sync-openapi.yml) is used to trigger the automation of the GitHub Apps documentation. The workflow runs automatically on a schedule. The workflow that triggers the GitHub Apps pipeline also triggers other automation pipelines that use the OpenAPI as the source data:
+- GitHub Apps
+- REST
+- Webhooks
+
+The workflow automatically creates a pull request with the changes (for all three pipelines) and the label `github-openapi-bot`.
+
+The workflow runs the `src/rest/scripts/update-files.js` script.
+
+## Local development
+
+To run the GitHub Apps pipeline locally:
+
+1. Clone the [`github/rest-api-description`](https://github.com/github/rest-api-description) repository inside your local `docs-internal` repository. 
+1. Run `src/rest/scripts/update-files.js -s rest-api-description -o github-apps`.
+
+## About this directory
+
+- `src/rest/data` - The automatically generated data files created by running this pipeline.
+- `src/rest/lib` - The source code used in production for the automated documentation generated by the GitHub Apps pipeline and configuration files edited by content and engineering team members.
+  - `src/rest/lib/config.json` - A configuration file used to specify metadata about the GitHub Apps pipeline. This file contains the SHA of the commit in `github/rest-api-description` that was used to generate the data in `src/github-apps/data`. This value is automatically updated when the pipeline runs.
+- `src/rest/scripts` - The scripts and source code used run the GitHub Apps pipeline, which updates the `src/github-apps/data` directory. 
+- `src/rest/tests` - The tests used to verify the GitHub Apps pipeline.
+
+## Configuring the pipeline
+
+The `src/github-apps/lib/config.json` file can contain any metadata needed by the content and engineering teams to configure the webhooks pipeline. The file currently only contains the following property:
+
+- `sha` - The SHA of the commit in `github/rest-api-description` that was used to generate the data in `src/github-apps/data`. This value is automatically updated when the pipeline runs.
+
+## Content team
+
+The content writers can manually update parts of the autogenerated Markdown file `content/rest/overview/endpoints-available-for-github-apps.md`. All frontmatter properties except `versions` can be manually modified and will not be overwritten on subsequent runs of the pipeline. The pipeline does update the `versions` property on each run.
+
+Writers can also add an introduction paragraph _above_ the following Markdown comment:
+
+```markdown
+<!-- Content after this section is automatically generated -->
+```
+
 ## How to get help
+
+Slack: `#docs-engineering`
+Repo: `github/docs-engineering`
+
+If you have a question about the GitHub Apps pipeline, you can ask in the `#docs-engineering` Slack channel. If you notice a problem with the GitHub Apps pipeline, you can open an issue in the `github/docs-engineering` repository.

src/rest/README.md

@@ -1,25 +1,71 @@
-# REST
+# REST pipeline
+
+Our REST pipeline creates autogenerated REST API documentation for docs.github.com/rest from the OpenAPI stored in the open-source repository [`github/rest-api-description`](https://github.com/github/rest-api-description).
+
+The pipeline is used to generate data that is used by the docs.github.com site when deployed locally, in preview environments, or in production.
+
+## How does it work
+
+![A flow chart describing how the automation pipeline for REST generates documentation](./rest-pipeline-flowchart.png)
+
+A [workflow](.github/workflows/sync-openapi.yml) is used to trigger the automation of the REST documentation. The workflow runs automatically on a schedule. The workflow that triggers the REST pipeline also triggers other automation pipelines that use the OpenAPI as the source data:
+- GitHub Apps
+- REST
+- Webhooks
+
+The workflow automatically creates a pull request with the changes (for all three pipelines) and the label `github-openapi-bot`. The workflow runs the `src/rest/scripts/update-files.js` script, which creates, deletes, or updates Markdown files in the `content/rest` directory.
+
+### Triggering the workflow sooner than the scheduled time
+
+You can manually run the workflow leaving the source branch input parameter with the default value of `main`. Running the workflow creates a new pull request in the `github/docs-internal` repo with the changes.
+
+### Triggering the workflow with a different source branch
+
+If there is a pull request that is still unmerged in `github/rest-api-description` that needs to be propagated to the docs.github.com site, manually run the workflow and provide the branch name of the pull request in the `github/rest-api-description` repo that you would like to get the changes from. This will create a new pull request in the `github/docs-internal` repo with the changes.
+
+## Local development
+
+To run the REST pipeline locally:
+
+1. Clone the [`github/rest-api-description`](https://github.com/github/rest-api-description) repository inside your local `docs-internal` repository. 
+1. Run `src/rest/scripts/update-files.js -s rest-api-description -o rest`. Note, by default `-o rest` is specified, so you can omit it.
 
 ## About this directory
 
-* `src/rest/lib/index.js` is human-editable.
-* `src/rest/data/*.json` are generated by [scripts](../../src/rest/scripts/README.md).
+- `src/rest/api` - The source code for the api endpoint used in production that redirects specific anchor links on a page under `docs.github.com/rest`.
+- `src/rest/data` - The automatically generated data files created by running this pipeline.
+- `src/rest/lib` - The source code used in production for the automated documentation generated by the REST pipeline and configuration files edited by content and engineering team members.
+  - `src/rest/lib/config.json` - A configuration file used to specify metadata about the REST pipeline.
+- `src/rest/scripts` - The scripts and source code used run the REST pipeline, which updates the `src/rest/data` directory. 
+  - `src/rest/scripts/update-files.js` - The entrypoint script that runs the REST pipeline.
+- `src/rest/tests` - The tests used to verify the REST pipeline.
 
-## Editable files
+## Configuring the pipeline
 
-* `src/rest/lib/index.js` consumes the static decorated schema files and exports `categories`, `operations`, and `operationsEnabledForGitHubApps` used by the REST middleware contextualizer.
+The `src/rest/lib/config.json` file contain metadata used by the content and engineering team to configure the REST pipeline. The file contains the following properties:
 
-## Static files
+- `api-versions` - A list of the calendar-date API versions that are available for each version.
+- `versionMapping` - A mapping of the API version name used in the OpenAPI schema in `github/rest-api-description` to the version short name used in the `github/docs-internal` repo.
+- `frontmatterDefaults` - A list of default frontmatter values to append to each Markdown file in `content/rest`.
+- `targetDirectory` - The directory in `content` where the Markdown files that correspond to this pipeline are located.
+- `indexOrder` - The order of the children in the `index.md` file in the `targetDirectory`. The `startsWith` keyword lists the directories under `targetDirectory` that should be listed first. All other automatically created directories are appended to this list.
+- `sha` - The SHA of the commit in `github/rest-api-description` that was used to generate the data in `src/rest/data` and Markdown files in `content/rest`. This value is automatically updated when the pipeline runs.
 
-Generated by `src/rest/scripts/update-files.js`:
+## Content team
 
-* `src/rest/data` - files generated from the dereferenced OpenAPI schema with the Markdown descriptions rendered in HTML
-* `src/rest/data/dereferenced` - **NOTE** These are only generated if you pass the --keep-dereferenced-files option and are not checked into the repository. This option is for debug only.
+The content writers can manually update parts of the autogenerated Markdown files in `content/rest`. When new Markdown files are added they will get all of the frontmatter properties defined in the `defaultFrontmatter` property in `src/rest/lib/config.js`. 
 
-## Rendering docs
+When a new Markdown file is created, a writer can manually change any of the frontmatter except `versions`. The pipeline will overwrite the `versions` property on subsequent runs, but will not modify any other frontmatter properties.
 
-When the server starts, `middleware/contextualizers/rest.js` accesses the data exported from the static decorated JSON files, fetches the data for the current version and requested path, and adds it to the `context` object. The added property is:
+Writers can also add an introduction paragraph _above_ the following Markdown comment:
 
-* `req.context.currentRestOperations` - all operations with a category matching the current path
+```markdown
+<!-- Content after this section is automatically generated -->
+```
 
-Markdown files in `content/rest/reference` use Liquid to loop over these context properties. The Liquid calls HTML files in the `includes` directory to do most of the rendering. Writers can add content to the Markdown files alongside the Liquid.
+## How to get help
+
+Slack: `#docs-engineering`
+Repo: `github/docs-engineering`
+
+If you have a question about the REST pipeline, you can ask in the `#docs-engineering` Slack channel. If you notice a problem with the REST pipeline, you can open an issue in the `github/docs-engineering` repository.

src/webhooks/README.md

@@ -1,44 +1,51 @@
-# Webhooks
+# Webhooks pipeline
+
+Our webhooks pipeline creates autogenerated webhooks documentation for docs.github.com/webhooks-and-events/webhooks/webhook-events-and-payloads from the OpenAPI stored in the open-source repository [`github/rest-api-description`](https://github.com/github/rest-api-description).
+
+The pipeline is used to generate data that is used by the docs.github.com site when deployed locally, in preview environments, or in production.
+
+## How does it work
+
+![A flow chart describing how the automation pipeline for webhooks generates documentation](./webhooks-pipeline-flowchart.png)
+
+A [workflow](.github/workflows/sync-openapi.yml) is used to trigger the automation of the webhooks documentation. The workflow runs automatically on a schedule. The workflow that triggers the webhooks pipeline also triggers other automation pipelines that use the OpenAPI as the source data:
+- GitHub Apps
+- REST
+- Webhooks
+
+The workflow automatically creates a pull request with the changes (for all three pipelines) and the label `github-openapi-bot`.
+
+The workflow runs the `src/rest/scripts/update-files.js` script, which then calls the `src/webhooks/scripts/sync.js` script.
+
+## Local development
+
+To run the webhooks pipeline locally:
+
+1. Clone the [`github/rest-api-description`](https://github.com/github/rest-api-description) repository inside your local `docs-internal` repository. 
+1. Run `src/rest/scripts/update-files.js -s rest-api-description -o webhooks`.
 
 ## About this directory
 
-* `src/webhooks/lib/index.js` is human-editable.
-* `src/rest/data/**/*.payload.json` are manually edited and copied. When a new GHES release is created, the static webhook files from the previous version's directory are copied to a new version directory.
+- `src/webhooks/data` - The automatically generated data files created by running this pipeline.
+- `src/webhooks/lib` - The source code used in production to display the webhook docs and configuration files edited by content and engineering team members.
+  - `src/webhooks/lib/config.json` - A configuration file used to specify metadata about the webhooks pipeline.
+- `src/webhooks/scripts` - The scripts and source code used run the webhooks pipeline, which updates the `src/webhooks/data` directory. 
+  - `src/webhooks/scripts/sync.js` - The entrypoint script that runs the webhooks pipeline.
+- `src/webhooks/tests` - The tests used to verify the webhooks pipeline.
 
-## Editable files
+## Configuring the pipeline
 
-* `src/webhooks/lib/index.js` consumes the static JSON files in `src/webhooks/data` and exports the data used by the REST middleware contextualizer.
+The `src/webhooks/lib/config.json` file can contain any metadata needed by the content and engineering teams to configure the webhooks pipeline. The file currently only contains the following property:
 
-## Static files
+- `sha` - The SHA of the commit in `github/rest-api-description` that was used to generate the data in `src/webhooks/data`. This value is automatically updated when the pipeline runs.
 
-Generated by `src/rest/scripts/update-files.js`:
+## Content team
 
-* `src/rest/data` - files generated from the dereferenced OpenAPI schema with the Markdown descriptions rendered in HTML
-* `src/rest/data/dereferenced` - **NOTE** These are only generated if you pass the --keep-dereferenced-files option and are not checked into the repository. This option is for debug only.
+The content writers can manually update frontmatter and introductory content in the associated Markdown file `content/webhooks-and-events/webhooks/webhook-events-and-payloads`. Automated content gets appended to the end of the file.
 
-## Rendering docs
+## How to get help
 
-When the server starts, `middleware/contextualizers/webhooks.js` accesses the data exported from the static webhook JSON files, fetches the data for the current version and requested path, and adds it to the `context` object. The added property is:
+Slack: `#docs-engineering`
+Repo: `github/docs-engineering`
 
-* `req.context.webhookPayloadsForCurrentVersion` - all webhook payloads with a version matching the current version
-
-Markdown files in `content/developers/webhooks-and-events/webhooks/webhook-events-and-payloads.md` use Liquid to display the webhook payloads in `req.context.webhookPayloadsForCurrentVersion`. For example `{{ webhookPayloadsForCurrentVersion.user.created }}` references the payload file `user.created.payload.json` for the version being viewed.
-
-**Note** Payload files either contain the webhook action type or no action type at all. For example, `user.created.payload.json` is the webhook `user` with the action type of `created`. Not all webhooks have action types. If a file exists with no action type (e.g., `user.payload.json`) and the action types (e.g., `user.created.payload.json` and `user.deleted.payload.json`), the entry in the context for the file with no action type will be `default`. For example, for the three static file mentioned, the object would be:
-
-```
-{
-  user: {
-    default: "STRING VALUE",
-    created: "STRING VALUE",
-    deleted: "STRING VALUE"
-  }
-}
-```
-
-If no action types exist, and only `user.payload.json` exists, the object would be:
-
-```
-{
-  user: "STRING VALUE"
-}
+If you have a question about the webhooks pipeline, you can ask in the `#docs-engineering` Slack channel. If you notice a problem with the webhooks pipeline, you can open an issue in the `github/docs-engineering` repository.

src/webhooks/scripts/sync.js

@@ -4,7 +4,7 @@ import path from 'path'
 import { mkdirp } from 'mkdirp'
 
 import { WEBHOOK_DATA_DIR, WEBHOOK_SCHEMA_FILENAME } from '../lib/index.js'
-import Webhook from '../lib/webhook.js'
+import Webhook from './webhook.js'
 
 export async function syncWebhookData(sourceDirectory, webhookSchemas) {
   await Promise.all(