docs: update sharding doc (#24558)
This commit is contained in:
Dmitry Gozman
GitHub
Родитель
8da37b364b
Коммит
29e63d1deb
|
|
@ -1,6 +1,6 @@
|
|||
---
|
||||
id: test-parallel
|
||||
title: "Parallelism and sharding"
|
||||
title: "Parallelism"
|
||||
---
|
||||
|
||||
Playwright Test runs tests in parallel. In order to achieve that, it runs several worker processes that run at the same time. By default, **test files** are run in parallel. Tests in a single file are run in order, in the same worker process.
|
||||
|
|
@ -125,16 +125,13 @@ test('runs second', async () => {
|
|||
|
||||
## Shard tests between multiple machines
|
||||
|
||||
Playwright Test can shard a test suite, so that it can be executed on multiple machines. For that, pass `--shard=x/y` to the command line. For example, to split the suite into three shards, each running one third of the tests:
|
||||
Playwright Test can shard a test suite, so that it can be executed on multiple machines.
|
||||
See [sharding guide](./test-sharding.md) for more details.
|
||||
|
||||
```bash
|
||||
npx playwright test --shard=1/3
|
||||
npx playwright test --shard=2/3
|
||||
npx playwright test --shard=3/3
|
||||
```
|
||||
|
||||
That way your test suite completes 3 times faster.
|
||||
|
||||
## Limit failures and fail fast
|
||||
|
||||
You can limit the number of failed tests in the whole test suite by setting `maxFailures` config option or passing `--max-failures` command line flag.
|
||||
|
|
|
|||
|
|
@ -1,117 +1,103 @@
|
|||
---
|
||||
id: test-sharding
|
||||
title: "Running tests on multiple machines"
|
||||
title: "Sharding"
|
||||
---
|
||||
|
||||
Playwright Test runs tests in [parallel](/test-parallel.md) and strives for optimal utilization of all accessible CPU cores on your machine. As the number of tests in your suite grows, you can improve execution speed by running them on several machines simultaneously.
|
||||
By default, Playwright runs tests in [parallel](/test-parallel.md) and strives for optimal utilization of CPU cores on your machine. In order to achieve even greater parallelisation, you can further scale Playwright test execution by running tests on multiple machines simultaneously. We call this mode of operation "sharding".
|
||||
|
||||
## Sharding tests between multiple machines
|
||||
|
||||
Playwright Test can shard a test suite, so that it can be executed on multiple machines. For that, pass `--shard=x/y` to the command line. For example, to split the suite into three shards, each running one third of the tests:
|
||||
To shard the test suite, pass `--shard=x/y` to the command line. For example, to split the suite into four shards, each running one fourth of the tests:
|
||||
|
||||
```bash
|
||||
npx playwright test --shard=1/3
|
||||
npx playwright test --shard=2/3
|
||||
npx playwright test --shard=3/3
|
||||
npx playwright test --shard=1/4
|
||||
npx playwright test --shard=2/4
|
||||
npx playwright test --shard=3/4
|
||||
npx playwright test --shard=4/4
|
||||
```
|
||||
|
||||
Now, if you run these shards in parallel on different computers, your test suite completes three times faster.
|
||||
Now, if you run these shards in parallel on different computers, your test suite completes four times faster.
|
||||
|
||||
### GitHub Actions sharding example
|
||||
## Merging reports from multiple shards
|
||||
|
||||
One of the easiest ways to shard Playwright tests across multiple machines is by using GitHub Actions matrix strategy. For example, you can configure a job to run your tests on 4 machines in parallel like this:
|
||||
In the previous example, each test shard has its own test report. If you want to have a combined report showing all the test results from all the shards, you can merge them.
|
||||
|
||||
Start with adding `blob` reporter to the config when running on CI:
|
||||
|
||||
```ts title="playwright.config.ts"
|
||||
export default defineConfig({
|
||||
testDir: './tests',
|
||||
reporter: process.env.CI ? 'blob' : 'html',
|
||||
});
|
||||
```
|
||||
|
||||
Blob report contains information about all the tests that were run and their results as well as all test attachments such as traces and screenshot diffs. Blob reports can be merged and converted to any other Playwright report. By default, blob report will be generated into `blob-report` directory.
|
||||
|
||||
To merge reports from multiple shards, put the blob report files into a single directory, for example `all-blob-reports`, and run `merge-reports` tool:
|
||||
|
||||
```bash
|
||||
npx playwright merge-reports ./all-blob-reports --reporter html
|
||||
```
|
||||
|
||||
This will produce a standard HTML report into `playwright-report` directory.
|
||||
|
||||
## GitHub Actions example
|
||||
|
||||
One of the easiest ways to shard Playwright tests across multiple machines is by using GitHub Actions matrix strategy. For example, you can configure a job to run your tests on four machines in parallel like this:
|
||||
|
||||
```yaml
|
||||
jobs:
|
||||
test:
|
||||
playwright-tests:
|
||||
strategy:
|
||||
fail-fast: false
|
||||
matrix:
|
||||
shard: [1/4, 2/4, 3/4, 4/4]
|
||||
shard: [1, 2, 3, 4]
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- uses: actions/checkout@v3
|
||||
- uses: actions/setup-node@v3
|
||||
with:
|
||||
node-version: 18
|
||||
- name: Install dependencies
|
||||
run: npm ci
|
||||
- name: Install Playwright browsers
|
||||
run: npx playwright install
|
||||
|
||||
- name: Run Playwright tests
|
||||
run: npx playwright test --shard ${{ matrix.shard }}
|
||||
run: npx playwright test --shard ${{ matrix.shard }}/4
|
||||
|
||||
- name: Upload HTML report
|
||||
uses: actions/upload-artifact@v3
|
||||
with:
|
||||
name: html-report-${{ matrix.shard }}
|
||||
path: playwright-report
|
||||
```
|
||||
|
||||
## Creating combined report in GitHub Actions
|
||||
|
||||
In the previous example, each test shard will have its own test report. If you want to have a combined report showing all tests results from all shards, you need to add a separate job that will merge individual shard reports.
|
||||
|
||||
### Configuring shard reporting
|
||||
|
||||
We start by adding `blob` reporter to the config:
|
||||
|
||||
```js
|
||||
export default defineConfig({
|
||||
testDir: './tests',
|
||||
reporter: [['blob', { outputDir: 'blob-report' }]],
|
||||
});
|
||||
```
|
||||
|
||||
Blob report contains information about all the tests that were run and their results as well as all test attachments such as traces and screenshot diffs. Blob reports can be merged and converted to any other Playwright report.
|
||||
|
||||
### Uploading shard report
|
||||
|
||||
To merge individual reports they need to be copied into a shared location. GitHub Actions Artifacts is a convenient mechanism that lets you do that:
|
||||
|
||||
```yaml
|
||||
jobs:
|
||||
test:
|
||||
...
|
||||
- name: Upload blob report to Artifacts
|
||||
- name: Upload blob report to GitHub Actions Artifacts
|
||||
if: always()
|
||||
uses: actions/upload-artifact@v3
|
||||
with:
|
||||
name: blob-report-${{ github.run_attempt }}
|
||||
path: blob-report
|
||||
retention-days: 2
|
||||
retention-days: 1
|
||||
```
|
||||
|
||||
By adding this step after the test execution we upload blob report from each shard into GitHub Actions Artifact with name`blob-report-${{ github.run_attempt }}` (this is essentially a shared directory where each shard will copy its report to). Note that instead of uploading individual HTML reports, we upload blob report.
|
||||
|
||||
### Merging reports
|
||||
|
||||
After all shards finished running it's time to run a job that will merge the reports and produce a combined HTML report. To ensure the execution order, we make `merge-report` job [depend](https://docs.github.com/en/actions/using-jobs/using-jobs-in-a-workflow#defining-prerequisite-jobs) on our sharded `test` job:
|
||||
After all shards have completed, run a separate job that will merge the reports and produce a combined HTML report.
|
||||
|
||||
```yaml
|
||||
jobs:
|
||||
merge-report:
|
||||
...
|
||||
merge-reports:
|
||||
# Merge reports after playwright-tests, even if some shards have failed
|
||||
if: always()
|
||||
needs: [test]
|
||||
needs: [playwright-tests]
|
||||
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- uses: actions/checkout@v3
|
||||
- uses: actions/setup-node@v3
|
||||
with:
|
||||
node-version: 18
|
||||
- name: Install dependencies
|
||||
run: npm ci
|
||||
|
||||
- name: Download Blob Reports from Artifacts
|
||||
- name: Download blob reports from GitHub Actions Artifacts
|
||||
uses: actions/download-artifact@v3
|
||||
with:
|
||||
name: blob-report-${{ github.run_attempt }}
|
||||
path: blob-report
|
||||
path: all-blob-reports
|
||||
|
||||
- name: Merge into HTML Report
|
||||
run: |
|
||||
npx playwright merge-reports ./blob-report --reporter html
|
||||
run: npx playwright merge-reports ./all-blob-reports --reporter html
|
||||
|
||||
- name: Upload HTML report
|
||||
uses: actions/upload-artifact@v3
|
||||
|
|
@ -120,45 +106,53 @@ jobs:
|
|||
path: playwright-report
|
||||
```
|
||||
|
||||
`merge-report` job above reads all blob reports from `blob-report-${{ github.run_attempt }}` artifact and produces a single HTML report. The job will run even if there were test failures and it wil write the HTML report into `playwright-report` directory by default.
|
||||
To ensure the execution order, we make `merge-reports` job [depend](https://docs.github.com/en/actions/using-jobs/using-jobs-in-a-workflow#defining-prerequisite-jobs) on our sharded `playwright-tests` job.
|
||||
|
||||
### Serving report: GitHub Actions Artifacts
|
||||
## Publishing report on the web
|
||||
|
||||
In the example above, we upload the HTML report as GitHub Actions Artifact. This is easy to configure, but downloading HTML report as a zip file is not very convenient.
|
||||
In the previous example, the HTML report is uploaded to GitHub Actions Artifacts. This is easy to configure, but downloading HTML report as a zip file is not very convenient.
|
||||
|
||||
In the next section, we'll illustrate how to make the report accessible from cloud storage.
|
||||
We can utilize Azure Storage's static websites hosting capabilities to easily and efficiently serve HTML reports on the Internet, requiring minimal configuration.
|
||||
|
||||
### Serving report: Azure Blob Storage
|
||||
1. Create an [Azure Storage account](https://learn.microsoft.com/en-us/azure/storage/common/storage-account-create).
|
||||
1. Enable [Static website hosting](https://learn.microsoft.com/en-us/azure/storage/blobs/storage-blob-static-website-how-to#enable-static-website-hosting) for the storage account.
|
||||
1. Add the Azure connection string as a [GitHub Actions secret](https://docs.github.com/en/actions/security-guides/encrypted-secrets#creating-encrypted-secrets-for-a-repository) called `AZURE_CONNECTION_STRING`.
|
||||
1. Add a step that uploads HTML report to Azure Storage.
|
||||
|
||||
We can utilize Azure Storage's [Static websites hosting](https://learn.microsoft.com/en-us/azure/storage/blobs/storage-blob-static-website) capabilities to easily and efficiently serve HTML reports on the Internet, requiring minimal configuration. You can simply add a step that uploads HTML report to Azure:
|
||||
```yaml
|
||||
...
|
||||
- name: Upload HTML report to Azure
|
||||
shell: bash
|
||||
run: |
|
||||
REPORT_DIR='run-${{ github.run_id }}-${{ github.run_attempt }}'
|
||||
az storage blob upload-batch -s playwright-report -d "\$web/$REPORT_DIR" --connection-string "${{ secrets.AZURE_CONNECTION_STRING }}"
|
||||
```
|
||||
|
||||
```yaml
|
||||
- name: Upload HTML report to Azure
|
||||
shell: bash
|
||||
run: |
|
||||
REPORT_DIR='run-${{ github.run_id }}-${{ github.run_attempt }}'
|
||||
az storage blob upload-batch -s playwright-report -d "\$web/$REPORT_DIR" --connection-string "${{ secrets.AZURE_CONNECTION_STRING }}"
|
||||
```
|
||||
|
||||
The code above assumes that you have the Azure connection string stored in GitHub [repository secret](https://docs.github.com/en/actions/security-guides/encrypted-secrets#creating-encrypted-secrets-for-a-repository) called `AZURE_CONNECTION_STRING`.
|
||||
|
||||
Afrer you enable [static website hosting](https://learn.microsoft.com/en-us/azure/storage/blobs/storage-blob-static-website#setting-up-a-static-website) for your storage account, the contents of `$web` can be accessed from a browser by using the public URL of the website ([how to find the website URL](https://learn.microsoft.com/en-us/azure/storage/blobs/storage-blob-static-website-how-to?tabs=azure-portal#portal-find-url)).
|
||||
The contents of `$web` storage container can be accessed from a browser by using the [public URL](https://learn.microsoft.com/en-us/azure/storage/blobs/storage-blob-static-website-how-to?tabs=azure-portal#portal-find-url) of the website.
|
||||
|
||||
:::note
|
||||
Note that this step will not work for pull requests created from a forked repository because such workflow [does't have access to the secrets](https://docs.github.com/en/actions/security-guides/encrypted-secrets#using-encrypted-secrets-in-a-workflow)
|
||||
This step will not work for pull requests created from a forked repository because such workflow [doesn't have access to the secrets](https://docs.github.com/en/actions/security-guides/encrypted-secrets#using-encrypted-secrets-in-a-workflow).
|
||||
:::
|
||||
|
||||
### Serving report: other CI and storage systems
|
||||
## Merge-reports CLI
|
||||
|
||||
At the high level, running multiple shards and generating a single combined report requires:
|
||||
`npx playwright merge-reports path/to/blob-reports-dir` reads all blob reports from the passed directory and merges them into a single report.
|
||||
|
||||
1. Configure Playwright to produce `blob` report on every running shard.
|
||||
1. Copy all blob reports into a single local directory.
|
||||
1. Run `npx playwright merge-reports path/to/all-blob-reports-dir --reporter html` to generate HTML (or any other) report.
|
||||
1. Upload generated report to the storage of your choice.
|
||||
Supported options:
|
||||
- `--reporter reporter-to-use`
|
||||
|
||||
Similarly to the GitHub Actions steps above, you can integrate these steps into your CI.
|
||||
Which report to produce. Can be multiple reporters separated by comma.
|
||||
|
||||
### Uploading Pull Request reports
|
||||
Example: `npx playwright merge-reports ./blob-reports --reporter=html,github`
|
||||
|
||||
TODO
|
||||
- `--config path/to/config/file`
|
||||
|
||||
Takes reporters from Playwright configuration file.
|
||||
|
||||
Example: `npx playwright merge-reports ./blob-reports --config=merge.config.ts`
|
||||
|
||||
```ts title="merge.config.ts"
|
||||
export default {
|
||||
reporter: [['html', { open: 'never' }]],
|
||||
};
|
||||
```
|
||||
|
|
|
|||
Загрузка…
Ссылка в новой задаче