зеркало из https://github.com/github/docs.git
Merge pull request #16565 from github/build-indices-for-given-version
Build indices for given version or language
This commit is contained in:
Sarah Schneider
GitHub
Коммит
64fa3b8767
|
|
@ -0,0 +1,44 @@
|
|||
name: Algolia Sync Single English Index
|
||||
|
||||
on:
|
||||
pull_request:
|
||||
types:
|
||||
- opened
|
||||
- reopened
|
||||
- synchronize
|
||||
- ready_for_review
|
||||
- unlocked
|
||||
|
||||
# This workflow requires a label in the format `sync-english-index-for-<PLAN@RELEASE>`
|
||||
jobs:
|
||||
updateIndices:
|
||||
name: Update English index for single version based on a label's version
|
||||
if: github.repository == 'github/docs-internal' && github.event.label.name.startsWith('sync-english-index-for-')
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- name: checkout
|
||||
uses: actions/checkout@5a4ac9002d0be2fb38bd78e4b4dbde5606d7042f
|
||||
- uses: actions/setup-node@56899e050abffc08c2b3b61f3ec6a79a9dc3223d
|
||||
with:
|
||||
node-version: 14.x
|
||||
- name: cache node modules
|
||||
uses: actions/cache@0781355a23dac32fd3bac414512f4b903437991a
|
||||
with:
|
||||
path: ~/.npm
|
||||
key: ${{ runner.os }}-node-${{ hashFiles('**/package-lock.json') }}
|
||||
restore-keys: |
|
||||
${{ runner.os }}-node-
|
||||
- name: npm ci
|
||||
run: npm ci
|
||||
- name: Get version from label
|
||||
id: getVersion
|
||||
run: |
|
||||
echo "::set-output name=version::$(github.event.label.name.split('sync-english-index-for-')[1])"
|
||||
- name: Sync English index for single version
|
||||
env:
|
||||
VERSION: ${{ steps.getVersion.outputs.version }}
|
||||
LANGUAGE: 'en'
|
||||
ALGOLIA_APPLICATION_ID: ${{ secrets.ALGOLIA_APPLICATION_ID }}
|
||||
ALGOLIA_API_KEY: ${{ secrets.ALGOLIA_API_KEY }}
|
||||
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
||||
run: npm run sync-search
|
||||
|
|
@ -1,4 +1,18 @@
|
|||
## Search
|
||||
# Search
|
||||
|
||||
## Table of contents
|
||||
- [Overview](#overview)
|
||||
- [Production deploys](#production-deploys)
|
||||
- [Manual sync from a checkout](#manual-sync-from-a-checkout)
|
||||
- [Build without sync (dry run)](#build-without-sync-dry-run)
|
||||
- [Build and sync](#build-and-sync)
|
||||
- [Label-triggered Actions workflow](#label-triggered-actions-workflow)
|
||||
- [Files](#files)
|
||||
- [GitHub Actions workflow files](#github-actions-workflow-files)
|
||||
- [Code files](#code-files)
|
||||
|
||||
|
||||
## Overview
|
||||
|
||||
This site's search functionality is powered by [Algolia](https://www.algolia.com), a third-party service.
|
||||
|
||||
|
|
@ -10,30 +24,77 @@ To see all existing search-related issues and pull requests, visit [github.com/g
|
|||
|
||||
---
|
||||
|
||||
## How it Works
|
||||
## Production deploys
|
||||
|
||||
The search data is synced automatically using a [GitHub Actions workflow](.github/workflows/sync-algolia-search-indices.yml) that is triggered by pushes to the `main` branch. This process generates structured data for all pages on the site, compares that data to what's currently on Algolia, then adds, updates, or removes indices based on the diff of the local and remote data, being careful not to create duplicate records and avoiding any unnecessary (and costly) indexing operations.
|
||||
A [GitHub Actions workflow](.github/workflows/sync-algolia-search-indices.yml) triggered by pushes to the `main` branch syncs the search data to Algolia. This process generates structured data for all pages on the site, compares that data to what's currently on Algolia, then adds, updates, or removes indices based on the diff of the local and remote data, being careful not to create duplicate records and avoiding any unnecessary (and costly) indexing operations.
|
||||
|
||||
The Actions workflow usually takes about five minutes, and the progress can be viewed (by GitHub employees) in the [Actions tab](https://github.com/github/docs/actions?query=workflow%3AAlgolia) of the repo.
|
||||
The Actions workflow progress can be viewed (by GitHub employees) in the [Actions tab](https://github.com/github/docs/actions?query=workflow%3AAlgolia) of the repo.
|
||||
|
||||
## Development
|
||||
Because the workflow runs after a branch is merged to `main`, there is a slight delay for search data updates to appear on the site.
|
||||
|
||||
In cases where a publicity event like GitHub Satellite or GitHub Universe demands a very tight shipping window, it is also possible to manually sync the indices with Algolia's servers from your local checkout of the repo, before your feature branch is merged to main. Manually syncing the indices can also be useful to test an unreleased GitHub Enterprise version or a translated language (Portuguese, Chinese, etc) that is not yet in production.
|
||||
## Manual sync from a checkout
|
||||
|
||||
To sync the indices from your development environment:
|
||||
It is also possible to manually sync the indices to Algolia from your local checkout of the repo, before your branch is merged to `main`.
|
||||
|
||||
1. Make sure the two required environment variables `ALGOLIA_APPLICATION_ID` and `ALGOLIA_API_KEY` are set in your `.env` file. These can be retrieved from the [Algolia site](https://www.algolia.com/apps/ZI5KPY1HBE/api-keys/all).
|
||||
2. Run `npm run sync-search-dry-run`. This takes a while to complete. It will prepare, test, and validate all the indices without actually uploading anything to Algolia's servers.
|
||||
3. Run `npm run sync-search` to prepare the indices again and upload them to the Algolia servers.
|
||||
**Prerequisite:** Make sure the environment variables `ALGOLIA_APPLICATION_ID` and `ALGOLIA_API_KEY` are set in your `.env` file. You can find these values on [Algolia](https://www.algolia.com/apps/ZI5KPY1HBE/api-keys/all).
|
||||
|
||||
### Build without sync (dry run)
|
||||
|
||||
To build all the indices without uploading them to Algolia's servers (this takes about an hour):
|
||||
```
|
||||
npm run sync-search-dry-run
|
||||
```
|
||||
To build indices for a specific language and/or version (this is much faster):
|
||||
```
|
||||
VERSION=<PLAN@RELEASE> LANGUAGE=<TWO-LETTER CODE> npm run sync-search-dry-run
|
||||
```
|
||||
You can set `VERSION` and `LANGUAGE` individually, too.
|
||||
|
||||
Substitute a currently supported version for `<PLAN@RELEASE>` and a currently supported two-letter language code for `<TWO-LETTER-CODE>`.
|
||||
|
||||
### Build and sync
|
||||
|
||||
To build all the indices and sync them to Algolia (this also takes about an hour):
|
||||
```
|
||||
npm run sync-search
|
||||
```
|
||||
To build indices for a specific language and/or version and sync them to Algolia:
|
||||
```
|
||||
VERSION=<PLAN@RELEASE LANGUAGE=<TWO-LETTER CODE> npm run sync-search
|
||||
```
|
||||
You can set `VERSION` and `LANGUAGE` individually, too.
|
||||
|
||||
Substitute a currently supported version for `<PLAN@RELEASE>` and a currently supported two-letter language code for `<TWO-LETTER-CODE>`.
|
||||
|
||||
## Label-triggered Actions workflow
|
||||
|
||||
Docs team members can use an Actions workflow on GHES release PRs by applying a label in this format:
|
||||
```
|
||||
sync-english-index-for-<PLAN@RELEASE>
|
||||
```
|
||||
This label will run a workflow on every push that **builds and uploads ONLY** the English index for the specified version. This means:
|
||||
|
||||
* The GHES content will be searchable at the same time the release PR is shipped, with no delay.
|
||||
* The GHES content will be searchable on staging throughout content creation.
|
||||
* No manual steps (unless you want to do a [dry run test](#build-without-sync-dry-run)).
|
||||
|
||||
Why do we need this? For our daily shipping needs, it's tolerable that search updates aren't available for up to an hour after the content goes live. But GHES releases are more time-sensitive, and writers have a greater need to preview search data on staging.
|
||||
|
||||
## Files
|
||||
|
||||
- [.github/workflows/sync-algolia-search-indices.yml](.github/workflows/sync-algolia-search-indices.yml) - the GitHub Actions workflow file that updates search indices whenever the main branch is updated.
|
||||
- [javascripts/search.js](javascripts/search.js) - the browser-side code that enables search using Algolia's [InstantSearch.js](https://github.com/algolia/instantsearch.js/) library.
|
||||
- [lib/algolia/client.js](lib/algolia/client.js) - a thin wrapper around the [algoliasearch](https://ghub.io/algoliasearch) Node.js module for interacting with the Algolia API.
|
||||
- [lib/algolia/search-index.js](lib/algolia/search-index.js) - a class for generating structured search data from repository content and syncing it with the remote Algolia service. This class has built-in validation to ensure that all records are valid before they're uploaded. This class also takes care of removing deprecated records, and compares existing remote records with the latest local records to avoid uploading records that haven't changed.
|
||||
- [script/sync-algolia-search-indices.js](script/sync-algolia-search-indices.js) - the script used by the Actions workflow to update search indices on our Algolia account. This can also be [run in the development environment](#development).
|
||||
- [tests/algolia-search.js](tests/algolia-search.js) - tests!
|
||||
### Actions workflow files
|
||||
|
||||
- [`.github/workflows/sync-algolia-search-indices.yml`](.github/workflows/sync-algolia-search-indices.yml) - Builds and syncs search indices whenever the `main` branch is pushed to (that is, on production deploys).
|
||||
- [`.github/workflows/dry-run-sync-algolia-search-indices.yml`](.github/workflows/dry-run-sync-algolia-search-indices.yml) - This workflow can be run manually (via `workflow_dispatch`) to do a dry run build of all the indices. Useful for confirming that the indices can build without erroring out.
|
||||
- [`.github/workflows/sync-single-english-algolia-index.yml`](.github/workflows/sync-single-english-algolia-index.yml) - This workflow is run when a label in the right format is applied to a PR. See "[Label-triggered Actions workflow](#label-triggered-actions-workflow)" for details.
|
||||
|
||||
### Code files
|
||||
|
||||
- [javascripts/search.js](javascripts/search.js) - The browser-side code that enables search using Algolia's [InstantSearch.js](https://github.com/algolia/instantsearch.js/) library.
|
||||
- [lib/algolia/client.js](lib/algolia/client.js) - A thin wrapper around the [algoliasearch](https://ghub.io/algoliasearch) Node.js module for interacting with the Algolia API.
|
||||
- [lib/algolia/search-index.js](lib/algolia/search-index.js) - A class for generating structured search data from repository content and syncing it with the remote Algolia service. This class has built-in validation to ensure that all records are valid before they're uploaded. This class also takes care of removing deprecated records, and compares existing remote records with the latest local records to avoid uploading records that haven't changed.
|
||||
- [script/sync-algolia-search-indices.js](script/sync-algolia-search-indices.js) - The script used by the Actions workflow to update search indices on our Algolia account. This can also be [run in the development environment](#development).
|
||||
- [tests/algolia-search.js](tests/algolia-search.js) - Tests!
|
||||
|
||||
## Indices
|
||||
|
||||
|
|
|
|||
|
|
@ -15,17 +15,43 @@ const allVersions = require('../all-versions')
|
|||
// e.g. `github-docs-dotcom-en.json` and `github-docs-2.14-ja.json`
|
||||
module.exports = async function syncAlgoliaIndices (opts = {}) {
|
||||
if (opts.dryRun) {
|
||||
console.log('This is a dry run! The script will abort without uploading anything.')
|
||||
console.log('This is a dry run! The script will build the indices locally but not upload anything.\n')
|
||||
rimraf(cacheDir)
|
||||
mkdirp(cacheDir)
|
||||
}
|
||||
|
||||
if (opts.language) {
|
||||
if (!Object.keys(languages).includes(opts.language)) {
|
||||
console.log(`Error! ${opts.language} not found. You must provide a currently supported two-letter language code.`)
|
||||
process.exit(1)
|
||||
}
|
||||
}
|
||||
|
||||
if (opts.version) {
|
||||
if (!Object.keys(allVersions).includes(opts.version)) {
|
||||
console.log(`Error! ${opts.version} not found. You must provide a currently supported version in <PLAN@RELEASE> format.`)
|
||||
process.exit(1)
|
||||
}
|
||||
}
|
||||
|
||||
// build indices for a specific language if provided; otherwise build indices for all languages
|
||||
const languagesToBuild = opts.language
|
||||
? Object.keys(languages).filter(language => language === opts.language)
|
||||
: Object.keys(languages)
|
||||
|
||||
// build indices for a specific version if provided; otherwise build indices for all veersions
|
||||
const versionsToBuild = opts.version
|
||||
? Object.keys(allVersions).filter(version => version === opts.version)
|
||||
: Object.keys(allVersions)
|
||||
|
||||
console.log(`Building indices for ${opts.language || 'all languages'} and ${opts.version || 'all versions'}.\n`)
|
||||
|
||||
// Exlude WIP pages, hidden pages, index pages, etc
|
||||
const indexablePages = await findIndexablePages()
|
||||
|
||||
// Build and validate all indices
|
||||
for (const languageCode in languages) {
|
||||
for (const pageVersion in allVersions) {
|
||||
for (const languageCode of languagesToBuild) {
|
||||
for (const pageVersion of versionsToBuild) {
|
||||
// if GHES, resolves to the release number like 2.21, 2.22, etc.
|
||||
// if FPT, resolves to 'dotcom'
|
||||
// if GHAE, resolves to 'ghae'
|
||||
|
|
|
|||
|
|
@ -14,7 +14,9 @@ main()
|
|||
async function main () {
|
||||
const sync = require('../lib/algolia/sync')
|
||||
const opts = {
|
||||
dryRun: 'DRY_RUN' in process.env
|
||||
dryRun: 'DRY_RUN' in process.env,
|
||||
language: process.env.LANGUAGE,
|
||||
version: process.env.VERSION
|
||||
}
|
||||
await sync(opts)
|
||||
}
|
||||
|
|
|
|||
Загрузка…
Ссылка в новой задаче