As part of this change, the confusingly named global variable 'state' is
renamed to 'currentURLTargetType', and named "enum" values are assigned
to it rather than raw integers.
MozReview-Commit-ID: FTEOB9wF8Q1
If the reftest harness times out the load of a URL before the 'load' event
has even fired then the error message that we get is 'load failed with unknown
reason'. This isn't very helpful for people unfamiliar with the harness. This
change sets an initial error message that notes that we're waiting on the
'load' event, what the timeout delay is, and what URL we are/were waiting on
loading. This will allow the timeout delay to be compared to log timestamps
and will make it clear whether the timeout occurred for some other URL than
the one we were expecting to load (which would be an error in the harness
logic).
It should now be impossible for the 'load failed with unknown reason' to occur,
but if there is a logic error in the harness code (some race condition?) then
it may still happen.
MozReview-Commit-ID: JOb1kYBpLro
Minor note:
reftests should've turned off uploadEnabled in the first place.
reftests should have unified telemetry on. It's the future.
MozReview-Commit-ID: 9spzuUAXwwP
This requires exposing the reftest chrome package to content, but that should be
OK, and this seems to work...
MozReview-Commit-ID: EWkwqTHW3dg
--HG--
extra : rebase_source : 12cabe4389375ac4c3abd0a9327baf268aab7c1a
Right now, e10s defaults to "on" in Nightly via the default pref
browser.tabs.remote.autostart.2=true. The reftest harness ignores that pref
and attempts to enable it by setting browser.tabs.remote.autostart to true (as
well as disabling the checks for non-mpc-compatible extensions). There's a
bug, however, and we overwrite the value for .autostart by reading
reftest-prefernces.js (which unconditionally sets .autostart to false).
Therefore, the existing code works because it ignores the .autostart.2 pref
and whether we enable e10s is entirely controlled by setting
extensions.e10sBlocksEnabling (the reftest extension is not mpc-compatible).
With this change, we unconditionally set .remote to the value that we want
(and we trust that the only non-mpc-compatible extensions are specially vetted
and part of our build system) so that things work as advertised on the box.
MozReview-Commit-ID: Li5N4NP4PwD
--HG--
extra : rebase_source : 50f842bcf6a13e8daeae8c0c76696c2edeb8f96b
There are two motivations for this change.
First, reftest.jsm has become very large and monolithic. It has lots of global
state and is hard to modify without breaking something. This change is a first
attempt at dividing reftest.jsm into multiple standalone(ish) modules. This
will make it easier to comprehend and extend.
Second, we'd like to implement "run-by-manifest" mode for reftest. This means
we'll restart the browser between each manifest run. This means much of the
state which is normally stored in global variables in reftest.jsm, will instead
need to be stored in python. A prerequisite to doing that is parsing the
manifests before starting the test suite. A prerequisite to that is moving the
manifest parsing code into a standalone JSM. This is the first step.
MozReview-Commit-ID: 42epTs8EU1O
--HG--
extra : rebase_source : 3db27445af0969867857fbfe41add13161a01cb8
(Path is actually r=froydnj.)
Bug 1400459 devirtualized nsIAtom so that it is no longer a subclass of
nsISupports. This means that nsAtom is now a better name for it than nsIAtom.
MozReview-Commit-ID: 91U22X2NydP
--HG--
rename : xpcom/ds/nsIAtom.h => xpcom/ds/nsAtom.h
extra : rebase_source : ac3e904a21b8b48e74534fff964f1623ee937c67
On Windows log lines are buffered in 4kb chunks instead of line buffered. This
means when a log on stdout exceeds this limit, a log from stderr can be
interleaved in the middle. This was causing frequent intermittent failures on
Windows when logging the suite_start message (which is much larger than 4kb).
This patch ensures that instead of redirecting stderr to stdout, we process
that stream independtly (though still using the same output handler). This
means we are guaranteed not to have any log interleaving, but it comes at the
cost of potentially losing the true log ordering. For that reason, this
behaviour is only enabled on Windows. Only the ordering between streams can be
different, and in practice this difference should be really small.
There is currently no good solution for both separating stdout/stderr and
preserving exact log ordering. See bug 798300 for more details.
MozReview-Commit-ID: 5W8I4u15uyM
--HG--
extra : rebase_source : ff6ce214b738dd09081b4b359a49ea7c3b0e4f65
The main bug this fixes is that on reftest, the objdir needs to be added to the
whitelist on Windows. However, this only happens when running on Linux for some
reason.
Changing the --work-path and --obj-path arguments to --sandbox-read-whitelist
was more of a drive-by cleanup than anything necessary.
MozReview-Commit-ID: Dq8ZLETMzeM
--HG--
extra : rebase_source : 3d2cdda125205e76f86235eb373074899fe0789a
If Firefox crashes while mochitests or reftests are running, Marionette
will trigger an IOError exception which currently gets logged immediately,
and causes no post-test checks to be performed. This results in missing
crash and leak checks, and an unclear failure message on Treeherder.
With this change only the IOError from Marionette gets deferred until
all post-test checks are done. This fixes the failure messages, and will
put PROCESS_CRASH or leak log output first.
MozReview-Commit-ID: JCYP5LlPE1m
--HG--
extra : rebase_source : a4a9455402b01db8ef1dbafccc7a726d2927ec03
Currently reftest.jsm uses logger.testEnd(..) to log all kinds of different
potential failures, from proper test failures to harness issues. This means
there are all kinds of edge cases that would cause multiple testEnd messages to
get logged, something the structured log protocol is supposed to prevent.
This modifies the reftest harness to instead use testStatus for everything. The
testEnd call will always be logged with status "OK". This required some changes
to the reftest formatter and reftest selftests.
MozReview-Commit-ID: 8RxsmHW49oy
--HG--
extra : rebase_source : 2f53e63f00bfc5e0e751b0f2bb3dc67477dc07eb
This adds new test verification steps for mochitest/reftest/xpcshell tests
with MOZ_CHAOSMODE=3 (thread scheduling and network thread scheduling).
Enabling all chaos mode features seems to destabilize test verification
so I am only enabling these features for now.
The tests that were run using this UI were removed in bug 652192. (They
did before-patch vs. after-patch comparisons of frame dumps, and didn't
actually have pass/fail state.)
MozReview-Commit-ID: 5qoeaFpEyQg
This installs specialpowers in the test runs so we can trigger crashes/assertions.
MozReview-Commit-ID: 8878OcHv8hU
--HG--
rename : testing/mochitest/tests/python/files/leaks.log => layout/tools/reftest/selftest/files/leaks.log
extra : rebase_source : cee417db30ff76364fa3212a65b2d537c97cc8bd
Complications:
- had to copy ReftestManifest into a test zip
- reftest harness was emitting multiple suite_start log entries with --repeat
- some extra path manipulation required to find reftests
The test plugins - dom/plugins/test/testplugin - are not built on Android.
By assuming that the test plugin is not available on Android, a per-manifest
check and warning can be avoided.
Currently the mochitest and reftest runners are forcing a timeout of 60s
(or 900s for valgrind or debug builds) when calling `start_session` of
Marionette. While this method still offers a timeout parameter, the
timeout should be set via the `startup_timeout` argument for Marionette.
Reason is that Marionette doesn't control the browser instance and
is getting called right after the application gets launched. As such
slow running builds can cause timeouts once it takes longer than 60s
for the Marionette server to get started.
By using `startup_timeout` the timeout will even be configurable via
the command line and mozharness config files.
MozReview-Commit-ID: EV7GklBcJjU
--HG--
extra : rebase_source : 9d3c623c49deb92d68c40ba4410c812c864e06bd
This just adds two basic tests, one for a passing test and another for a
failing one. In mochitest, we use privileged APIs to also tests crashes,
assertions, asan and leaks. But these APIs aren't available to reftests
so I'm not sure how we can test these things.
I figure it's not worth holding the framework up on this though, I'll file
a follow-up to figure out something to do for that.
MozReview-Commit-ID: 59TSbsugT5T
--HG--
extra : rebase_source : 72ecd817017c8b7d55eab879db4f6ad5fecc54c0
Skip tests that are expected to fail in both Stylo and Gecko modes. They would unexpectedly "pass" in styloVsGecko mode when comparing the two failures, which is not a useful result.
MozReview-Commit-ID: 3mOpjU225Q1
--HG--
extra : rebase_source : 22bb5d4e3c5138ef832995eaf5716824f4707ffe
extra : source : d40fb20c9a49d0797c0eeae613a04912b12a28f7
This renames killNamedOrphans to killNamedProc and removes the check for
parent proc id, so that any xpcshell or ssltunnel process is killed before
starting a mochitest or reftest run. For reftests, this is moved out of
the desktop harness and into the remote/android harness, since desktop
reftests do not use xpcshell or ssltunnel.
Since these tests harnesses are already disabling Safe Browsing and tracking
protection, they should also disable the other two features that cause
lists to be downloaded from the Mozilla shavar server.
MozReview-Commit-ID: 2158qRU4XZx
--HG--
extra : rebase_source : 287bfb7580538cca7f58ab035b7141166346c318
The FrameLoaderOwner interface has been implemented in WebIDL for several
years now, so these QIs are simply unnecessary overhead.
MozReview-Commit-ID: LAzvfm5Qhy0
--HG--
extra : rebase_source : 2495c07df21c474f5fabc257ff4db43b0d8047e4
This is straightforward, with only two notable things.
- `#include "nsXPIDLString.h" is replaced with `#include "nsString.h"`
throughout, because all nsXPIDLString.h did was include nsString.h. The
exception is for files which already include nsString.h, in which case the
patch just removes the nsXPIDLString.h inclusion.
- The patch removes the |xpidl_string| gtest, but improves the |voided| test to
cover some of its ground, e.g. testing Adopt(nullptr).
--HG--
extra : rebase_source : 452cc4a08046a1adb1a8099a7e85a1917de5add8
The login reputation checks depend on a server lookup and therefore would
render non-deterministic the performance and correctness of tests.
MozReview-Commit-ID: Bil0rSZsGPT
--HG--
extra : rebase_source : ae06a028c71eee323307ecd4e62bbf1e8a14fe13
The login reputation checks depend on a server lookup and therefore would
render non-deterministic the performance and correctness of tests.
MozReview-Commit-ID: Bil0rSZsGPT
--HG--
extra : rebase_source : c9a641e4bf49c48bf864ed546bf2ae6eb51c27e4
This shouldn't be at module level.
This isn't technically needed for this series. I caught it during
an earlier attempt at teasing out all Python import dependencies
to run `mach`. Why let a good patch go to waste.
MozReview-Commit-ID: FwwdZqcKtpq
--HG--
extra : rebase_source : a8ce2cf452dc8c03371dc384ea4f1be9195b5d30
This mechanically replaces nsILocalFile with nsIFile in
*.js, *.jsm, *.sjs, *.html, *.xul, *.xml, and *.py.
MozReview-Commit-ID: 4ecl3RZhOwC
--HG--
extra : rebase_source : 412880ea27766118c38498d021331a3df6bccc70
This also guards against passing a non-test parameter to reftest.startAfter.
MozReview-Commit-ID: FoqqN4D7sv7
--HG--
extra : rebase_source : 0e5d879467c04803e3ac565b979aceaad3176597
This avoids mistakenly blaming the first reftest in the suite for a crash that
happens during startup. Blaming that reftest can result in fragmentation of the
resulting issues because any bugs filed by sheriffs will be test-specific instead
of grouped together into a single bug.
MozReview-Commit-ID: K4YelNzXxzg
--HG--
extra : rebase_source : b291f365956997bbc3d591addee6817ad3e0ff9b
Test harnesses may use STYLO_FORCE_ENABLED, so we need to check this value when
building the reftest conditions.
MozReview-Commit-ID: 998UMZHNKLl
--HG--
extra : rebase_source : c52e2539c0384c2c300b729ac88fe3381e127509
This adds commands to start a reftest session, run a test, and end the
session. It as assumed that after you start a reftest session you will
just run reftests until you end the session. When starting a session
the user provides a string indicating when screenshots should be
taken, and an object mapping urls to a count of the number of times
that url is expected to be used in the session, to help with
caching. Running the tests takes a url to a test, an expected status,
a timeout, and a nested list of possible references, in which each
entry at a specific level is combined by OR and nested references are
combined by AND.
The implementation is heavilly inspired by the existing reftest
harness, starting a minimal window with no tabs, and loading the urls
directly in there. In order to get a screenshot in the e10s case we
have to pass the DRAW_VIEW and USE_WIDGET_LAYERS flags when taking the
screenshot.
For performance we heavily cache canvases; for references that will be
repeated we cache the full canvas with image, and we also cache a
single canvas to use for all other screenshots to avoid the overhead
of repeatedly creating a new canvas element.
MozReview-Commit-ID: JOFvtmH7tg
This adds commands to start a reftest session, run a test, and end the
session. It as assumed that after you start a reftest session you will
just run reftests until you end the session. When starting a session
the user provides a string indicating when screenshots should be
taken, and an object mapping urls to a count of the number of times
that url is expected to be used in the session, to help with
caching. Running the tests takes a url to a test, an expected status,
a timeout, and a nested list of possible references, in which each
entry at a specific level is combined by OR and nested references are
combined by AND.
The implementation is heavilly inspired by the existing reftest
harness, starting a minimal window with no tabs, and loading the urls
directly in there. In order to get a screenshot in the e10s case we
have to pass the DRAW_VIEW and USE_WIDGET_LAYERS flags when taking the
screenshot.
For performance we heavily cache canvases; for references that will be
repeated we cache the full canvas with image, and we also cache a
single canvas to use for all other screenshots to avoid the overhead
of repeatedly creating a new canvas element.
MozReview-Commit-ID: JOFvtmH7tg
--HG--
extra : rebase_source : ab5a2ef2e450b9bbdc6bc3c9487ed5dfda2c1d4b
This is needed before we can upgrade to flake8 3.3.0, as that version starts flagging these errors.
These files were modified by running:
autopep8 --select E305 --in-place -r <dir>
on the affected directories. I did it one dir at a time and verified the result after each.
MozReview-Commit-ID: FmlsfiKIbtr
--HG--
extra : rebase_source : 9df32258cadff5d27a0e72113c57f782756c0b18
This won't be an issue until the 57 release but at that time, the
reftest extensions will only load if the extensions.legacy.enabled
preference is set to true and Cu.isInAutomation, which depends on
another preference, is true. Set the appropriate prefs here for reftests.
MozReview-Commit-ID: JUWCPTsuJU
--HG--
extra : rebase_source : 042a5696d1926e7ca434a7c4213395176a080549
This catches a common problem where somebody adds a fuzzy annotation on
a test to work around some minor differences. Later the differences go
away, but since the test harness doesn't catch that, nobody is the
wiser. Subsequently a "real" regression can reintroduce differences
which are hidden by the stale fuzzy annotations.
With this patch, if the annotations are set up properly, the test
harness will flag tests as "UNEXPECTED-PASS" when the differences go
away. This will require the patch author to reduce the allowed fuzziness
parameters, and will make it easier to catch subsequent regressions.
MozReview-Commit-ID: B3rGPFLXkCu
This patch extends the "fuzzy" and "fuzzy-if" annotations so that they
accept a range of fuzziness values, much like the "asserts" and
"asserts-if" annotations. If the test produces differences that are
within the specified range, the test is considered as passing. Any
differences outside the ranges are considered failures. If the test is
marked as a "!=" test, then the opposite is true.
If the range given to "fuzzy" or "fuzzy-if" is a single value, it is
interpreted as being a range from zero to the specified value. This is
in contrast to "asserts" and "asserts-if" which match exactly the value
specified. The reason for this is mostly for backwards compatibility
with existing reftest annotation semantics.
MozReview-Commit-ID: 6qUU6FQ5mYP
This simply augments the logging in the reftest harness so that if a reftest
passes due to fuzzy matching, it reports both the actual fuzz numbers and the
maximum allowed fuzz numbers.
MozReview-Commit-ID: G356vBRXYUc
Remove the forbiddenURI pref which was removed in bug 1274893 as well
as browser.safebrowsing.enabled which got renamed in bug 1025965.
Set dummy URLs for all of the network endpoints.
MozReview-Commit-ID: Efk2fv6cC3g
--HG--
extra : rebase_source : 9fbb3eb0fa7f002fe24577a8a0870ec4d1b7cf31
Our linux32-debug build is very slow to startup when running mochitests on aws.
Sometimes we see similar behavior on other linux platforms. Intermittently,
in this environment, startup takes longer than the 120 seconds that marionette
waits, resulting in test failures in bug 1261598. Increasing the marionette
startup timeout to 180 seconds appears to effectively avoid these failures.
Remove suggested and enhanced tiles along with related campaign, frequency-cap, inadjacency, pings, preferences, strings, styles, tests.
MozReview-Commit-ID: FkjaSpSFQHu
--HG--
extra : rebase_source : 1c58ac542180f0abb290639ec1c61b9edf3d0a51
This prefix can't be produced by the standard structured logging
formatters because it doesn't know that the tests are
reftests. Therefore the reftest harness has a hack to add this prefix,
and the unstructured analyzer doesn't work with wpt reftests. This can
simply be solved by making the prefix optional, but looking for the
"INFO - " prefix that mozharness adds, when present, to identify the
start of the TEST- string.
MozReview-Commit-ID: GxWVWOSkAUZ
--HG--
extra : rebase_source : fbd8afa860da0fba4925b6bf2fc715cbf6c2688c
Non-ascii characters in error messages can cause exceptions when processing
structured log messages. The mochitest MessageLogger already handles such
cases; this is a simpler implementation for the reftest OutputHandler.
Previously the harness would check for navigator.plugins[Test Plug-in] in the browser window, but that is now blocked from working. Instead we can simply use an existing getTestPlugin function that enumerates the plugin through nsIPluginHost
MozReview-Commit-ID: Kp48u5iFkSa
Continue to allow non-multiprocessCompatible extensions in automation.
There are a ton of places that would need to be changed, many of which
will be changing soon anyway with the non-webextensions change in 57
so this is mostly the expedient route to keeping the tree green.
MozReview-Commit-ID: EZZoDVdhLfy
--HG--
extra : rebase_source : f83472bc1c88dd0deadbe485d9002499027ff07f
Continue to allow non-multiprocessCompatible extensions in automation.
There are a ton of places that would need to be changed, many of which
will be changing soon anyway with the non-webextensions change in 57
so this is mostly the expedient route to keeping the tree green.
MozReview-Commit-ID: EZZoDVdhLfy
--HG--
extra : rebase_source : 34aa762917566b052ade6372280caed72fbfbe9a
Ed Lee
Geoff Brown
Jonathan Watt
Chris H-C
Phil Ringnalda
Emilio Cobos Álvarez
Henrik Skupin
Masatoshi Kimura
Matt Woodrow ext:(%2C%20Miko%20Mynttinen%20%3Cmikokm%40gmail.com%3E%2C%20Timothy%20Nikkel%20%3Ctnikkel%40gmail.com%3E)
Blake Kaplan
Sebastian Hengst
Andrew McCreight
Andrew Halberstadt
Alex Gaynor
Tobias Schneider
Nicholas Nethercote
Nathan Froyd
Marina Samuel
Henri Sivonen
Kartikaya Gupta
L. David Baron
Chris Peterson
Gian-Carlo Pascutto
Kris Maglione
J. Ryan Stinnett
Francois Marier
Wes Kocher
Gregory Szorc
Bob Owen
k88hudson
Carsten "Tomcat" Book
Dan Mosedale ext:(%3E)
Samael Wang
Jonathan Kew
Haik Aftandilian
Sylvestre Ledru
Thomas Nguyen
James Graham
David Anderson
Olli Pettay
Xidorn Quan
Dão Gottwald
Andrew Swan
Shing Lyu
Mike Cooper
Bobby Holley
Benjamin Smedberg