As part of this change, the confusingly named global variable 'state' is
renamed to 'currentURLTargetType', and named "enum" values are assigned
to it rather than raw integers.
MozReview-Commit-ID: FTEOB9wF8Q1
If the reftest harness times out the load of a URL before the 'load' event
has even fired then the error message that we get is 'load failed with unknown
reason'. This isn't very helpful for people unfamiliar with the harness. This
change sets an initial error message that notes that we're waiting on the
'load' event, what the timeout delay is, and what URL we are/were waiting on
loading. This will allow the timeout delay to be compared to log timestamps
and will make it clear whether the timeout occurred for some other URL than
the one we were expecting to load (which would be an error in the harness
logic).
It should now be impossible for the 'load failed with unknown reason' to occur,
but if there is a logic error in the harness code (some race condition?) then
it may still happen.
MozReview-Commit-ID: JOb1kYBpLro
There are two motivations for this change.
First, reftest.jsm has become very large and monolithic. It has lots of global
state and is hard to modify without breaking something. This change is a first
attempt at dividing reftest.jsm into multiple standalone(ish) modules. This
will make it easier to comprehend and extend.
Second, we'd like to implement "run-by-manifest" mode for reftest. This means
we'll restart the browser between each manifest run. This means much of the
state which is normally stored in global variables in reftest.jsm, will instead
need to be stored in python. A prerequisite to doing that is parsing the
manifests before starting the test suite. A prerequisite to that is moving the
manifest parsing code into a standalone JSM. This is the first step.
MozReview-Commit-ID: 42epTs8EU1O
--HG--
extra : rebase_source : 3db27445af0969867857fbfe41add13161a01cb8
Currently reftest.jsm uses logger.testEnd(..) to log all kinds of different
potential failures, from proper test failures to harness issues. This means
there are all kinds of edge cases that would cause multiple testEnd messages to
get logged, something the structured log protocol is supposed to prevent.
This modifies the reftest harness to instead use testStatus for everything. The
testEnd call will always be logged with status "OK". This required some changes
to the reftest formatter and reftest selftests.
MozReview-Commit-ID: 8RxsmHW49oy
--HG--
extra : rebase_source : 2f53e63f00bfc5e0e751b0f2bb3dc67477dc07eb
Complications:
- had to copy ReftestManifest into a test zip
- reftest harness was emitting multiple suite_start log entries with --repeat
- some extra path manipulation required to find reftests
The test plugins - dom/plugins/test/testplugin - are not built on Android.
By assuming that the test plugin is not available on Android, a per-manifest
check and warning can be avoided.
Skip tests that are expected to fail in both Stylo and Gecko modes. They would unexpectedly "pass" in styloVsGecko mode when comparing the two failures, which is not a useful result.
MozReview-Commit-ID: 3mOpjU225Q1
--HG--
extra : rebase_source : 22bb5d4e3c5138ef832995eaf5716824f4707ffe
extra : source : d40fb20c9a49d0797c0eeae613a04912b12a28f7
The FrameLoaderOwner interface has been implemented in WebIDL for several
years now, so these QIs are simply unnecessary overhead.
MozReview-Commit-ID: LAzvfm5Qhy0
--HG--
extra : rebase_source : 2495c07df21c474f5fabc257ff4db43b0d8047e4
This avoids mistakenly blaming the first reftest in the suite for a crash that
happens during startup. Blaming that reftest can result in fragmentation of the
resulting issues because any bugs filed by sheriffs will be test-specific instead
of grouped together into a single bug.
MozReview-Commit-ID: K4YelNzXxzg
--HG--
extra : rebase_source : b291f365956997bbc3d591addee6817ad3e0ff9b
Test harnesses may use STYLO_FORCE_ENABLED, so we need to check this value when
building the reftest conditions.
MozReview-Commit-ID: 998UMZHNKLl
--HG--
extra : rebase_source : c52e2539c0384c2c300b729ac88fe3381e127509
This catches a common problem where somebody adds a fuzzy annotation on
a test to work around some minor differences. Later the differences go
away, but since the test harness doesn't catch that, nobody is the
wiser. Subsequently a "real" regression can reintroduce differences
which are hidden by the stale fuzzy annotations.
With this patch, if the annotations are set up properly, the test
harness will flag tests as "UNEXPECTED-PASS" when the differences go
away. This will require the patch author to reduce the allowed fuzziness
parameters, and will make it easier to catch subsequent regressions.
MozReview-Commit-ID: B3rGPFLXkCu
This patch extends the "fuzzy" and "fuzzy-if" annotations so that they
accept a range of fuzziness values, much like the "asserts" and
"asserts-if" annotations. If the test produces differences that are
within the specified range, the test is considered as passing. Any
differences outside the ranges are considered failures. If the test is
marked as a "!=" test, then the opposite is true.
If the range given to "fuzzy" or "fuzzy-if" is a single value, it is
interpreted as being a range from zero to the specified value. This is
in contrast to "asserts" and "asserts-if" which match exactly the value
specified. The reason for this is mostly for backwards compatibility
with existing reftest annotation semantics.
MozReview-Commit-ID: 6qUU6FQ5mYP
This simply augments the logging in the reftest harness so that if a reftest
passes due to fuzzy matching, it reports both the actual fuzz numbers and the
maximum allowed fuzz numbers.
MozReview-Commit-ID: G356vBRXYUc
Previously the harness would check for navigator.plugins[Test Plug-in] in the browser window, but that is now blocked from working. Instead we can simply use an existing getTestPlugin function that enumerates the plugin through nsIPluginHost
MozReview-Commit-ID: Kp48u5iFkSa
Jonathan Watt
Noemi Erli
Matt Woodrow ext:(%2C%20Miko%20Mynttinen%20%3Cmikokm%40gmail.com%3E%2C%20Timothy%20Nikkel%20%3Ctnikkel%40gmail.com%3E)
Andrew Halberstadt
Tobias Schneider
Geoff Brown
Nathan Froyd
Sebastian Hengst
Henri Sivonen
Kartikaya Gupta
Chris Peterson
Kris Maglione
J. Ryan Stinnett
David Anderson
Carsten "Tomcat" Book
Shing Lyu
Bobby Holley
Benjamin Smedberg
Florian Queze