Two options for benchmarking:
Using the Test Runner
When working on issues related to performance, it is important to provide a performance benchmark for your changes to assess whether or not your change has a performance impact. PDF.js provides tools to do this easily. Normally you would create a simple manifest file has a couple of PDF files you're trying to optimize and run it multiple times, e.g. benchmark.json:
[
{
"id": "tracemonkey-eq",
"file": "pdfs/tracemonkey.pdf",
"md5": "9a192d8b1a7dc652a19835f6f08098bd",
"rounds": 50,
"lastPage": 5,
"type": "eq"
}
]
Run the following commands to create a 'baseline' measurement (i.e., before you make your changes):
$ git checkout master
$ npx gulp generic
$ npx gulp components
$ cd test
$ node test.mjs --statsFile=stats/results/baseline.json --statsDelay=5000 --manifestFile=benchmark.json
Then apply your changes and create a 'current' measurement (replace <feature-branch> with the name of your branch):
$ cd ..
$ git checkout <feature-branch>
$ npx gulp generic
$ npx gulp components
$ cd test
$ node test.mjs --statsFile=stats/results/current.json --statsDelay=5000 --manifestFile=benchmark.json
Finally, you can compare the measurements and see any performance differences:
$ node stats/statcmp.js stats/results/baseline.json stats/results/current.json
As a sanity check, you should do this twice with the same code and compare the results.
For faster benchmarking during development, it could be useful to pass the --noChrome command line parameter to test.js to skip Chrome and only test with Firefox.
Using Firefox Talos
About
Versions of PDF.js that are merged in to Firefox are automatically benchmarked as part of the talos tests to monitor performance regressions/improvements. The talos pdfpaint test benchmarks how long it takes from a PDF being opened to the first page of the PDF being rendered. The advantage of using this system for benchmarking is that it is closer to a real world benchmark where we test the most important thing, which is how long it takes for the first page to render.
Running Locally
Prerequisites
This assumes you've setup a local PDF.js repo and you've setup a build of Firefox locally. We recommend using an non-debug artifact build of Firefox.
Adding a Custom PDF
If you just want to benchmark your changes with the default PDFs you can skip this step.
- Copy the PDF file into
<firefox_src_dir>/testing/talos/talos/tests/pdfpaint/ - Add the file to the manifest in
<firefox_src_dir>/testing/talos/talos/tests/pdfpaint/pdfpaint.manifest - (Optional) Remove any other PDF files in the manifest if you only want to benchmark your PDF
Record the Baseline Data
First, we want to run the test PDF without any changes to see how it will compare. In the <firefox_src_dir> run:
./mach talos-test -a pdfpaint
When the test finishes there will be a section in the terminal with ------- Summary: start ------- and -------- Summary: end --------. You'll want to copy that section down somewhere for comparison.
Apply Your Changes & Test Again
- In the
<pdfjs_src_dir>:- make your changes
- run
npx gulp mozcentral - copy the pdf.js build to firefox
rsync -a -v <pdfjs_src_dir>/build/mozcentral/browser/extensions/pdfjs/ <firefox_src_dir>toolkit/components/pdfjs/
- Back in
<firefox_src_dir>:- Build firefox
./mach build faster - Run the tests again
./mach talos-test -a pdfpaint
- Build firefox
The talos tests will run and output a test summary. You can now compare these results to see how your changes affected performance.
Running pdfpaint on Mozilla's infrastructure
Similar to above, you'll want to create a baseline result without your changes and another result with your changes. The pdfpaint test is part of the talos "other" group. An example of running the tests 6 times on windows:
./mach try fuzzy -q 'windows10-64-shippable opt-talos-other' --rebuild 6 --artifact
Once the tasks finish, you can then compare the results by using the two commit IDs and replacing <baseline> and <changed> below:
https://treeherder.mozilla.org/perfherder/compare?originalProject=try&originalRevision=<baseline>&newProject=try&newRevision=<changed>&framework=1&showOnlyImportant=1&page=1