This patch fixes various issues that prevented us from running chromium/chrom in raptor-browsertime.
(1) Chromium fetch task now also fetches the latest chromedriver.
(2) FFMPEG failures when recording on chrome/chromium.
(3) Various changes where chromium wasn't considered as a variant of chrome.
Differential Revision: https://phabricator.services.mozilla.com/D92646
This patch enabled mitm5 for browsertime desktop tests. It also fixes a small issue in mozproxy which sets the return code to 0 on a failure.
Differential Revision: https://phabricator.services.mozilla.com/D92434
This patch makes raptor emit an error when the test being run is a browsertime test and the --browsertime flag wasn't provided.
Differential Revision: https://phabricator.services.mozilla.com/D91207
Since this bug, Firefox will start to use the same conversion as
Chrome for reporting FCP. This patch cleans our code a little bit
to adapt that.
Differential Revision: https://phabricator.services.mozilla.com/D90109
This patch adds the idle and idle-bg tests to raptor-browsertime. To make these work, a new browsertime script was built which handles waiting on a given page url, as well as backgrounding the application.
Differential Revision: https://phabricator.services.mozilla.com/D88491
This patch adds the ability to measure power usage from browsertime in raptor. The method used for gathering/storing power usage information here is different from raptor. In this case, we report the power usage information within the same perfherder-data blob as the technical information and it will all be saved within a single perfherder-data.json file.
Differential Revision: https://phabricator.services.mozilla.com/D88490
This patch adds a check for the verbose option in the OutputHandler. If it is set, then the log won't be filtered based on the presence of raptor in the logs.
Differential Revision: https://phabricator.services.mozilla.com/D83984
This patch adds a new pageload testing mode called chimera mode. This mode runs the cold and warm pageload tests together within a single browser session (cold first, warm second). Some changes had to be made in the visual metrics task to be able to do this (per-test specification of extraOptions).
Differential Revision: https://phabricator.services.mozilla.com/D83356
This patch adds all desktop pageload tests to browsertime. It adds taskcluster tasks for these but doesn't enable them in production. They can only be run when --full is used.
Differential Revision: https://phabricator.services.mozilla.com/D82432
This patch adds an option to disable the firefox window recorder in browsertime, it also adds it to the browsertime-mobile taskcluster config since it is disabled there (but not on desktop).
Differential Revision: https://phabricator.services.mozilla.com/D82431
This patch modifies how the extraOptions field in the resulting Raptor perfherder data is built. Instead of being spread across all over the file, we use a single function to fill the extra options. It can be customized as well (which we needed for some compatibility for Raptor-webext).
Differential Revision: https://phabricator.services.mozilla.com/D81595
Remove the BinAST tests from following jobs:
- SM(p)
- SM(cgc)
Remove the BinAST performance tests:
- binast-instagram
Mark the WPT BinAST tests as backlog.
Differential Revision: https://phabricator.services.mozilla.com/D77919
Remove the BinAST tests from following jobs:
- SM(p)
- SM(cgc)
Remove the BinAST performance tests:
- binast-instagram
Mark the WPT BinAST tests as backlog.
Differential Revision: https://phabricator.services.mozilla.com/D77919
This patch adds a class-variable to the RaptorLogger that will hold the application name being tested and it is only set when we test on Chrome. This will let us differentiate failure lines between Firefox and Chrome better, i.e. all failures without an app name in the line are Firefox failures, and otherwise they are Chrome failures.
Differential Revision: https://phabricator.services.mozilla.com/D79476
This patch disables and removes the raptor-webext fennec68 tests and replaces them with the browsertime variants. The raptor tests which ran through a cron are also swapped for the browsertime tests.
Differential Revision: https://phabricator.services.mozilla.com/D77674
The old raptor-youtube-playback test was converted to a 2020 branch H264 test and this patch reverts that change. It also adds a new test (and task) for the H264 standard tests.
Differential Revision: https://phabricator.services.mozilla.com/D78803
This patch prevents opening a new tab in fennec raptor tests to attempt to fix some intermittents we see in production.
Differential Revision: https://phabricator.services.mozilla.com/D76395
I need to add symbolication support for the mochitest Gecko Profiler command line
option. These profiles also need to be symbolicated. Unfortunately, there is not
a common place where I could use these files. Talos and Raptor each had their
own copy of the snappy symbolication server.
This commit consolidates these packages into a re-usable mozbase package that can
be used in mochitests, and eventually in other places like xpcshell tests.
I stubbed out a test file, but it doesn't do anything quite yet. This commit makes
it so that the tests still work in Raptor and Talos, but doesn't add any features.
It also doesn't try too hard to make the files look like a mozbase package.
Differential Revision: https://phabricator.services.mozilla.com/D74289
The current implementation opens the built Firefox in the objdir. This is not
optimal as the built Firefox is not really great for viewing files in. The build
could be broken. With this patch, the profiles will instead be opened in the users
default browser.
Differential Revision: https://phabricator.services.mozilla.com/D70089
Change --wasm-simd to --no-wasm-simd and flip the default for both this
switch and for the about:config flag.
Add testing directives so that we test baseline, ion, and --no-wasm-simd.
Differential Revision: https://phabricator.services.mozilla.com/D71819
This patch makes it so that we only download and extract the conditioned-profile once and have tests use copies of that original copy. It also adds some debugging logs to the condprof `download_file` function to debug an issue where we seem to hit a cache-miss when we have already downloaded the artifact.
Differential Revision: https://phabricator.services.mozilla.com/D75176
This patch reduces the redundancy in the mozproxy perfherder data names. It removes the `replay` suffix since it's already clear it's related to replay/not-replayed. The subtest names have the `mozproxy-replay` prefix removed through the addition of a new way of specifying supporting measurement values. Instead of only being able to submit the values with a `{measurement: values}` format, we can now submit it as `{measurement: {values: [], <SUBTEST-OPTIONS>}}`. The subtest options include perfherder-specific options, and some settings that change how the names are built. Furthermore, the full supporting data dictionary also includes two new fields: `summarize-values`, and `suffix-type`. The former dictates whether or not a summary value is produced, and the latter determines if the name should be suffixed with the data type.
Differential Revision: https://phabricator.services.mozilla.com/D72695
This patch adds the extra options settings to the visual metrics perfherder data by passing it through the jobs.json file.
Differential Revision: https://phabricator.services.mozilla.com/D73277
This patch disables conditioned-profiles in the cpu raptor tests - we have no need for conditioned profiles in this unittest.
Differential Revision: https://phabricator.services.mozilla.com/D73064
This patch adds a line handler to the mozprocess object responsible for running browsertime.
The line handler strips away some unnecessary information the log, and then outputs it using Raptor's log handler (level determined by browsertime log message). If a line that is prefixed with `error` we immediately raise an exception to kill the process handler, and then raise the failure in Raptor to prevent results processing.
Differential Revision: https://phabricator.services.mozilla.com/D72752
This requires adding a new JSOptions field (for internal use within the shell),
as well as a new browser pref (to support possible Cranelift benchmarking on
aarch64).
Differential Revision: https://phabricator.services.mozilla.com/D72907
Sleeping a second and increasing the elapsed time by 1,
doesn't mean that exactly a second has been passed-by.
For a better timeout handling the end time has to be
calculated first, and then checked against.
Differential Revision: https://phabricator.services.mozilla.com/D72132
- changed test URL to match the dev server
- changed output.py in several places to fix new test names, dict keys, to cover all tests
- added amazonaws.com to manifest.json file to fix the loading issue for benchmark.js file
- added all raptor tests
- changed the constants for measure and alert_on
Differential Revision: https://phabricator.services.mozilla.com/D62546
This patch changes how Google Chrome for Android is deployed. Rather than relying on automatic updates, we will use tooltool to download the APK and install it ourselves. Some changes were done in taskcluster to remove a hack that was put in place to disable interal tooltool downloads (the issue is resolved now).
A tooltool manifest is added for this, and to keep ourselves organized, all manifests (including the playback ones) are moved into a folder called `tooltool-manifests`.
Differential Revision: https://phabricator.services.mozilla.com/D72198
When Raptor moves Fennec into the background querying for
the active tab doesn't return any value. To fix that the
current tab id can be cached, which also doesn't require
us to always have to query for the current tab when updating
or closing it.
Differential Revision: https://phabricator.services.mozilla.com/D72033
This patch adds the new live site tests to the whitelist so that they can run. It also moves the playback settings back to where it was (before any test name changes) and adds some comments to make sure we're aware of this in the future.
Differential Revision: https://phabricator.services.mozilla.com/D72394
This patch adds some new live-site-only tests to Raptor-Browsertime, these will run on Fenix, & Chrome in CI but are enabled for all mobile browsers. The live sites reside in a new `live` folder since they can be used on desktop and mobile if needed.
Differential Revision: https://phabricator.services.mozilla.com/D69052
This is the first patch of a multi-part patch to enable live site testing on Google Chrome for Android and Fenix. It adds the `--live-sites` command line option that disables the mozproxy playback (or enables live sites) when it is supplied.
Differential Revision: https://phabricator.services.mozilla.com/D69051
Browsertime's -vvv displays the gecko profile passed through HTTP
in the logs, making them very big. This patch makes verbose output
an option and deactivates it by default.
Differential Revision: https://phabricator.services.mozilla.com/D71511
This patches fixes several problems found on Raptor and the condprof:
Raptor:
- Make sure the conditioned profile dir is removed after
it's been used, not before.
- Adds the --project option to raptor so we know if we're on try
autoland or mozilla-central.
- Both Fennec and Fenix are deactivated for now
- Use the allow-downgrade flag to be flexible on build ids (the next step will be bug 1628666)
Conditioned profiles, curation of the profile prefs:
- Fully deactivates Normandy during Raptor tests (app.normandy.enabled)
- Removes any GFX blacklisting (gfx.blacklist.*)
- Removes any marionette pref
- Enforce extensions sideloading (extensions.startupScanScopes)
Differential Revision: https://phabricator.services.mozilla.com/D70518
Some phones aren't updating their google-chrome version so we are hitting issues where the chromedriver is not being found because it's looking for an older version. This patch makes the error occur earlier and prevents browsertime from running if we hit it.
Differential Revision: https://phabricator.services.mozilla.com/D70210
--HG--
extra : moz-landing-system : lando
Activates the conditioned profile by doing the following changes:
- make sure the conditioned profile dir is removed after
it's been used, not before.
- add the --project option to raptor so we know if we're on try
or mozilla-central.
- Both Fennec and Fenix are deactivated for now.
- Remove any gfx.blacklist.* prefs when using a conditioned profile
Differential Revision: https://phabricator.services.mozilla.com/D70518
--HG--
extra : moz-landing-system : lando
Make sure the conditioned profile dir is removed after
it's been used, not before. This patch also adds the
--project option to raptor so we know if we're on try
or mozilla-central. Both Fennec and Fenix are deactivated
for now.
Differential Revision: https://phabricator.services.mozilla.com/D70518
--HG--
extra : moz-landing-system : lando
This patch changes the test that browsertime runs for speedometer from `raptor-speedometer-APP` to `speedometer` and changes some other small issues like the missing app name in the `extraOptions` field in the perfherder data.
Differential Revision: https://phabricator.services.mozilla.com/D69581
--HG--
extra : moz-landing-system : lando
There is no need to store the current testTabID, given
that we can always query for it. Also it would prevent
unforseen behavior in case of undefined or the custom
code for the default value 0.
Because Raptor always works with the current tab, lets
query for the current tab id whenever some tab action
is triggered.
Differential Revision: https://phabricator.services.mozilla.com/D69569
--HG--
extra : moz-landing-system : lando
Separates the checks for completion and the final clean-up steps
to simplify the code.
Also the same interval for checking for results is used across
all the test types now.
Differential Revision: https://phabricator.services.mozilla.com/D69568
--HG--
extra : moz-landing-system : lando
Various setTimeout usages to delay code execution causes multi-level
code stacking. By using the newly introduced sleep function, the code
can be streamlined and simplified a lot.
Using setTimeout also prevented us for correclty handling possible
Javascript errors, so that those weren't correctly propagated up the
stack. As such no webext_error gets send to the control server, which
should force an application shutdown. Instead Raptor would hang forever
until the application timeout happens.
Differential Revision: https://phabricator.services.mozilla.com/D69567
--HG--
extra : moz-landing-system : lando
This has been removed by bug 1620827 without any effect. As
such it has to be re-added to be consistent with Chrome.
Differential Revision: https://phabricator.services.mozilla.com/D69566
--HG--
extra : moz-landing-system : lando
For GeckoView vehicles no additional tab is created during the
initialization. As such one and only open tab cannot be closed.
Differential Revision: https://phabricator.services.mozilla.com/D69564
--HG--
extra : moz-landing-system : lando
To start the scenario timer as close as possible to the page
timeout alarm, it needs to be called before "collectResults".
Differential Revision: https://phabricator.services.mozilla.com/D69563
--HG--
extra : moz-landing-system : lando
When awaiting the promise to be resolved, the returned Tab object
will contain the neccessary reference to the id. As such no separate
listeners have to be defined. Side-effect with listeners is that those
are async and could cause race conditions in the sync program flow.
Differential Revision: https://phabricator.services.mozilla.com/D69562
--HG--
extra : moz-landing-system : lando
Not awaiting the promises to be resolved will cause race conditions,
which clearly lead to test failures.
Differential Revision: https://phabricator.services.mozilla.com/D69561
--HG--
extra : moz-landing-system : lando
There is no need to store the current testTabID, given
that we can always query for it. Also it would prevent
unforseen behavior in case of undefined or the custom
code for the default value 0.
Because Raptor always works with the current tab, lets
query for the current tab id whenever some tab action
is triggered.
Differential Revision: https://phabricator.services.mozilla.com/D69569
--HG--
extra : moz-landing-system : lando
Separates the checks for completion and the final clean-up steps
to simplify the code.
Also the same interval for checking for results is used across
all the test types now.
Differential Revision: https://phabricator.services.mozilla.com/D69568
--HG--
extra : moz-landing-system : lando
Various setTimeout usages to delay code execution causes multi-level
code stacking. By using the newly introduced sleep function, the code
can be streamlined and simplified a lot.
Using setTimeout also prevented us for correclty handling possible
Javascript errors, so that those weren't correctly propagated up the
stack. As such no webext_error gets send to the control server, which
should force an application shutdown. Instead Raptor would hang forever
until the application timeout happens.
Differential Revision: https://phabricator.services.mozilla.com/D69567
--HG--
extra : moz-landing-system : lando
This has been removed by bug 1620827 without any effect. As
such it has to be re-added to be consistent with Chrome.
Differential Revision: https://phabricator.services.mozilla.com/D69566
--HG--
extra : moz-landing-system : lando
For GeckoView vehicles no additional tab is created during the
initialization. As such one and only open tab cannot be closed.
Differential Revision: https://phabricator.services.mozilla.com/D69564
--HG--
extra : moz-landing-system : lando
To start the scenario timer as close as possible to the page
timeout alarm, it needs to be called before "collectResults".
Differential Revision: https://phabricator.services.mozilla.com/D69563
--HG--
extra : moz-landing-system : lando
When awaiting the promise to be resolved, the returned Tab object
will contain the neccessary reference to the id. As such no separate
listeners have to be defined. Side-effect with listeners is that those
are async and could cause race conditions in the sync program flow.
Differential Revision: https://phabricator.services.mozilla.com/D69562
--HG--
extra : moz-landing-system : lando
Not awaiting the promises to be resolved will cause race conditions,
which clearly lead to test failures.
Differential Revision: https://phabricator.services.mozilla.com/D69561
--HG--
extra : moz-landing-system : lando
This patch fixes an issue in warm pageload tests where the browser doesn't navigate to the page that was requested since the browser was already open on that page. To get around this, we navigate to about:blank, and then to the rquested url.
Differential Revision: https://phabricator.services.mozilla.com/D68428
--HG--
extra : moz-landing-system : lando
This patch adds the capability to run Google Chrome for Android tests through Raptor-Browsertime.
Chrome must be available on the device being tested, and there are no installation steps being added in this patch since CI already has the latest Chrome version available to test with. That said, this patch adds the ability to find the version of Chrome that is being tested and stores this in the Perfherder artifact. Getting this version is also necessary to be able to run Chrome with the correct chromedriver.
Two tests are initially be enabled for Chrome in this patch: Amazon, and YouTube. They will only run through a cron task three days a week. The other changes done in this patch are required for Chrome to work with Raptor-Browsertime.
Differential Revision: https://phabricator.services.mozilla.com/D65610
--HG--
extra : moz-landing-system : lando
This patch reduces the page load timeout for speedometer and unity-webgl tests to 7 minutes.
Differential Revision: https://phabricator.services.mozilla.com/D68059
--HG--
extra : moz-landing-system : lando
This patch fixes a bad URL that is currently being used for the google pageload tests in browsertime.
Differential Revision: https://phabricator.services.mozilla.com/D67919
--HG--
extra : moz-landing-system : lando
The method should never allow invalid settings for the
control server host and port to be written to the test
config as read by the webextension. It would mean that
it will never be able to send messages to the control
server, which will cause Raptor to kill the test with
an application timeout.
Differential Revision: https://phabricator.services.mozilla.com/D65450
--HG--
extra : moz-landing-system : lando
This patch removes a `storage.local.clear()` call from raptor-webext for causing a high number of intermittent failures. It also adds a default value for `--host` in the mozharness raptor. Finally, a delayed startup for raptorRunner is added with the hope that this will further decrease intermittents (right now, it might be starting at a very noisy time).
Differential Revision: https://phabricator.services.mozilla.com/D65893
--HG--
extra : moz-landing-system : lando
Major version number bump for both mozlog and mozcrash + requirements update.
This is very much like the previous mozlog major bump - see bug 1591384.
This patch does not update testing/web-platform/tests/tools/wptrunner/requirements.txt;
I expect that will be automatically updated by the pyup.io bot once mozlog is re-released.
Differential Revision: https://phabricator.services.mozilla.com/D65135
--HG--
extra : moz-landing-system : lando
Various code in the Raptor webextension that is written
asynchronously isn't used with 'await' or at least a
Promise chain. As result there will be overlapping
attempts to send messages to the control server that
can even vary in their order, and causing tests
to fail.
With that patch the amount of extra console logging
will be reduced to only those cases where sending the
message to the control server failed.
Differential Revision: https://phabricator.services.mozilla.com/D64348
--HG--
extra : moz-landing-system : lando
This patch adds some stability and comparison optimizations to the Browsertime arguments. It adds a viewport setting to ensure that all browsers we will test on use the same browser size (only affects desktop). The pageLoadStrategy is also set to `none` so that we can use the `pageCompleteCheckStartWait` and `pageCompleteCheckPollTimeout` options. Those options determine how much Selenium polls (and when it starts polling) to check if there are results available.
The size of the browser is also set to a specific size to prevent regressions/improvements (alerts) from occuring when we update Browsertime and the default screen resolution changes. This patch also disables the Browsertime Firefox extension since we don't need it.
Differential Revision: https://phabricator.services.mozilla.com/D63182
--HG--
extra : moz-landing-system : lando
This patch adds all Fenix pageload tests to raptor-browsertime. To handle how the task is specified (specifying run-on-projects by raptor-test) the raptor.py taskgraph transforms had to be modified.
The transform modifications change a couple things. First, the `raptor-subtests` is now the first thing to have it's keyed-by entries handled because it's values are used as keyed-by entries on other fields such as run-on-projects. Once it's values are resolved, we immediately split/clone the task into one per subtest entry (adding a `chunk-number` field that is used for more setup in the next transforms). After this, the rest of the fields have their keyed-by entries resolved.
Differential Revision: https://phabricator.services.mozilla.com/D63222
--HG--
extra : moz-landing-system : lando
This patch allows us to use the enable/disable charging commands for power tests, and moves the android power tests onto the primary perf bitbar devices. It also removes the use of the --host argument that was previously required to run power tests. Disabling charging occurs when the device is being setup, and when the device is being setup within raptor (to accomadate different entry points). Enabling charging occurs when the mach command finishes, or during the clean_up stage in raptor in normal, passing executions. In case any errors occur within raptor, and due to the multiple entry points, charging is enabled where those errors are caught as well.
Differential Revision: https://phabricator.services.mozilla.com/D42373
--HG--
extra : moz-landing-system : lando
This patch upgrades the major browsertime version used in-tree from 4 to 8 (including some additional fixes to fix some failing tests on our end).
We also add the node v10 requirement in this patch. Also, there were some changes in the browsertime repo's visualmetrics.py script that made it necessary to change where we find the file.
Differential Revision: https://phabricator.services.mozilla.com/D59235
--HG--
extra : moz-landing-system : lando
Gregory Mierzwinski
Razvan Maries
Mike Hommey
Florin Strugariu
Narcis Beleuzu
Ted Campbell
alexandru.ionescu
Kimberly Sereduck
Dorel Luca
Sean Feng
Bob Clary
Anny Gakhokidze
Mihai Alexandru Michis
Myeongjun Go
Csoregi Natalia
Tom Prince
Marian Raiciof
VandanRogheliya
Taegeon Lee
Mitchell Hentges
Greg Tatum
Lars T Hansen
Henrik Skupin
andraesanu
Benjamin Bouvier
Stefan Hindli
Tarek Ziadé
Tooru Fujisawa
octavian.negru
Paul Adenot
Brindusan Cristian
Brendan Dahl
Noemi Erli
Mark Banner
Geoff Brown
Bogdan Tara
Nick Alexander