This patch adds a new set of tests (essential only) to run in CI at tier 2. A command line argument is added to turn bytecode cache testing on in warm pageloads. This also disables the testing in our primary pageload tests to fix the variance and increases in runtime.
Differential Revision: https://phabricator.services.mozilla.com/D151757
Currently Raptor-Browsertime does not run with Python 3.10. Until then, an Exception that kills the Browsertime process with a message telling the user to downgrade their Python version would be useful.
Differential Revision: https://phabricator.services.mozilla.com/D151330
This patch addresses live login failures occuring in CI. It uses a similar implementation already in `perftest_record.js` to restrict login-logic to only happen on sites with Taskcluster secrets available. In the future we should improve this if/when more secrets are added and/or when we look into being able to easily share reusable code between Raptor and Mozperftest.
Differential Revision: https://phabricator.services.mozilla.com/D150623
This patch enables the collection of perfstats through raptor-browsertime. It can be enabled by setting `perfstats = true` in the test manifest.
Differential Revision: https://phabricator.services.mozilla.com/D148903
This patch adds some logic to clean up the artifact folder from the hundreds of PNG/JPEGs generated on task failures. It sorts the files and attempts to only keep the 5 newest images so that we can still use them for debugging issues.
Differential Revision: https://phabricator.services.mozilla.com/D148943
This first patch in this series ports the login logic from Bug 1742103 into Raptor-Browsertime so that we may now have automated login into websites during a test (e.g. live pageloads in CI).
As of present, there is no convenient way to share reusable code between Raptor-Browsertime and Mozperftest so a large majority of the code was copied over as is from `perftest_record.js` with a few more methods added.
Additional logic was also added (e.g. in `base.py`) to ensure that pageload tests executes as desired based on if the website requires a login or not, as defined by the `pageload_sites.json`
Differential Revision: https://phabricator.services.mozilla.com/D145926
This patch updates browsertime to add the following PR: https://github.com/sitespeedio/browsertime/pull/1798
It enables us to use a `--debug` flag in browsertime to pause test execution after each script iteration, and provides the ability for developers to pause it on their own by adding `debug.breakpoint(name)` calls to the test script. The execution can be unpaused by running `window.browsertime.pause = false` in the browser console.
The devtools panel is auto-opened when in debug-mode, and the browser is detached so it won't auto-close when you hit failures in raptor/browsertime.
This can only be used locally so we now raise a parser error when it's provided in CI.
Differential Revision: https://phabricator.services.mozilla.com/D148261
Currently gecko profiling is broken in chimera mode because of the recent path changes that weren't updated in the browsertime.json file or for other files in the file entry. This patch changes the logic to iterate through all of the files found in the files key (including gecko profiles).
Differential Revision: https://phabricator.services.mozilla.com/D147800
This patch fixes up how fission is handled in raptor. Previously, there were multiple ways in which it was handled, and now, we make all the decisions when we parse the command-line options. It also fixes the issue with fission being enabled on mobile tests.
Differential Revision: https://phabricator.services.mozilla.com/D147527
This patch causes us to trigger a rebuild for the python environment on browsertime upgrades and installations. This is needed for now because we are moving to python-based dependencies for visual-metrics processing and that requires a rebuild on the first upgrade/installation.
See bug 1766112 for the removal of this code.
Depends on D144464
Differential Revision: https://phabricator.services.mozilla.com/D144565
This patch forces us to re-install browsertime if FFMPEG is not found in the cache. This fixes an issue where deleting the `.mozbuild/browsertime` folder would go undetected and cause failures later when running browsertime.
Alongside these changes, the patch also enables detecting for FFMPEG on Linux, and Windows machines. It works in the same way as mac, but it also requires using `browsertime_ffmpeg` since the `os.environ` settings don't get transferred to the required environment.
Depends on D143603
Differential Revision: https://phabricator.services.mozilla.com/D144464
This patch runs the visual-metrics processing within the test task rather than the (deleted) `-vismet` tasks. It also updates the browsertime version to v15. It requires an update to node16 so new code was added to pull, and install the *-node-16 toolchain artifacts from CI.
This patch is possible because the browsertime update gives us the new `visualmetrics-portable.py` script which doesn't require ImageMagick anymore (they've been converted to Python dependencies). We did this conversion here: https://github.com/sitespeedio/browsertime/pull/1741
Furthermore, some changes are made to handle timeouts better, and increase the timeout for an interactive test.
Depends on D143502
Differential Revision: https://phabricator.services.mozilla.com/D142838
This patch adds the ability to raptor-browsertime for downloading, and installing a node binary from our Taskcluster CI. This makes it possible to seamlessly use raptor-browsertime with node-16, while using node-12 for the rest of the mozilla-central tooling.
Depends on D142837
Differential Revision: https://phabricator.services.mozilla.com/D143502
This patch builds off the ability to specify custom browsertime arguments on the command line in raptor to let the user run custom tests in "vanilla" browsertime. In this patch, we create a new test called `browsertime` that can be used to get a "dummy" template for a raptor test. Then, the user will be responsible for providing either a test script, or a URL to test.
To use a custom script we can pass something like `--browsertime-arg test_script=/path/to/script` or `--browsertime-arg url=https://www.sitespeed.io`. Furthermore, we can also use `test_script=pageload` to specify that we want to use the browsertime pageload script.
Differential Revision: https://phabricator.services.mozilla.com/D144168
This patch adds the ability to provide custom browsertime arguments through the command line option `--browsertime-arg`. It is used like so:
`
./mach raptor --browsertime -t browsertime --browsertime-arg test_script=/home/sparky/mozilla-source/mozilla-central/testing/raptor/browsertime/browsertime_pageload.js --browsertime-arg browsertime.url=https://www.sitespeed.io --browsertime-arg iterations=3
`
Differential Revision: https://phabricator.services.mozilla.com/D144166
The patch aims to streamline the local installation of visual metric packages on Intel based OSX.
The user first needs to run (assuming a fresh `./mach clobber && ./mach build`):
`./mach browsertime --clobber --setup --install-vismet-reqs`
then the user should be able to run vismet tests locally right away e.g.
`./mach raptor --browsertime --browsertime-visualmetrics -t wikipedia`
Differential Revision: https://phabricator.services.mozilla.com/D139889
This patch modifies condprof to accept an optional `remote_test_root` argument so that we can change where the profile gets created and pulled from. Then we pass the external test root through this parameter from raptor to condprof. This fixes an issue on the Samsung A51 device as it can't store the profiles in the default location.
Differential Revision: https://phabricator.services.mozilla.com/D139517
Myeongjun Go
Gregory Mierzwinski
Steve Fink
Kash Shampur
Alex Ionescu
Alex Finder
Molnar Sandor
eng_esther
andrej
Joel Maher
Marian-Vasile Laza
Florian Quèze
Henrik Skupin
Markus Stange