Currently the RefTest manifest parser has 5 possible statuses:
0 => EXPECTED_PASS
1 => EXPECTED_FAIL
2 => EXPECTED_RANDOM
3 => EXPECTED_DEATH (aka skip)
4 => EXPECTED_FUZZY
In the manifests, the last status annotation that appears on the line will take
precedence. For example:
skip-if(true) fails-if(true) == test1.html ref.html
fails-if(true) skip-if(true) == test2.html ref.html
The first test will have an expected status equal to EXPECTED_FAIL, whereas the
second one will be EXPECTED_DEATH. The same holds true for any combination of
'fail/random/skip/fuzzy' annotations. This means developers need to be very
careful about the order they specify these annotations as getting the order
wrong can easily lead to unexpected behaviour.
With the introduction of defaults in bug 1616368, the risk of unexpected behaviour
is far greater. Since defaults are simply prepended to the test line, a manifest
that looks like:
defaults skip-if(true)
== test1.html ref.html
fails-if(true) == test2.html ref.html
will actually skip the first test, but run the second (since the fails-if
overwrites EXPECTED_DEATH with EXPECTED_FAIL).
The root of the problem appears to be that 'skip' and 'fuzzy' are not actually
test statuses. They are modifiers that affect how we run the test, but don't
actually affect whether the test is expected to pass or fail.
Therefore, this patch solves the problem by making 'skip/skip-if' its own thing
that does not get overwritten by other failure types. In otherwords, a 'skip-if'
can appear before or after a 'fails-if' and it will have the same meaning.
Differential Revision: https://phabricator.services.mozilla.com/D64457
--HG--
extra : moz-landing-system : lando
With the new 'defaults' key being added, 'default-preferences' is now
redundant. This commit converts all existing uses of 'default-preferences' to
use 'defaults' instead.
Differential Revision: https://phabricator.services.mozilla.com/D63247
--HG--
extra : moz-landing-system : lando
This introduces a new 'defaults' key to reftest manifests. It works similarly
to the existing 'default-preferences' key except it can be used for any test
modifier (e.g, skip-if, fail-if, pref, etc.).
See the changes to README.txt in this commit for an explanation on how to use
it.
Differential Revision: https://phabricator.services.mozilla.com/D63246
--HG--
extra : moz-landing-system : lando
Use the manifest ID rather than the manifest path in the reftest suite_start message.
Differential Revision: https://phabricator.services.mozilla.com/D62081
--HG--
extra : moz-landing-system : lando
Use test identifiers that look like posix-compliant relative file paths rather than URLs; identifier "paths" are relative to topsrcdir -- an intuitive, brief mapping from identifier to test file.
This patch adds (expands) a --topsrcdir option to runreftest.py and updates CI configuration to set it appropriately. runreftest.py calculates a manifest ID, a path relative to topsrcdir, for each requested manifest and passes those manifest IDs to manifest.jsm. In manifest.jsm, a new manifest ID is generated when a manifest is included from a new directory. Test identifiers are generated by adding the test file name to the manifest ID.
Examples (url -> identifier):
- file:///builds/worker/workspace/build/tests/reftest/tests/dom/svg/crashtests/398926-fill.svg -> dom/svg/crashtests/398926-fill.svg
- file:///Z:/task_1577755655/build/tests/reftest/tests/dom/svg/crashtests/398926-fill.svg -> dom/svg/crashtests/398926-fill.svg
- http://10.7.205.207:8854/tests/dom/svg/crashtests/398926-fill.svg -> dom/svg/crashtests/398926-fill.svg
- file:///builds/worker/workspace/build/tests/jsreftest/tests/js/src/tests/jsreftest.html?test=test262/built-ins/Array/prototype/indexOf/15.4.4.14-9-b-i-31.js -> js/src/tests/test262/built-ins/Array/prototype/indexOf/15.4.4.14-9-b-i-31.js
- chrome://reftest/content/bugs/299837-2.xhtml -> layout/reftests/bugs/299837-2.xhtml
- http://localhost:36067/1577754972876/1/289480.html#top -> layout/reftests/bugs/289480.html#top
- about:blank -> about:blank
- data:text/html,<body> -> data:text/html,<body>
Differential Revision: https://phabricator.services.mozilla.com/D58415
--HG--
extra : moz-landing-system : lando
We enable the pref in the harness so that local testing with --enable-fission will get the same results as treeherder.
Differential Revision: https://phabricator.services.mozilla.com/D56313
--HG--
extra : moz-landing-system : lando
Adds a way for mochitest, reftest, and crashtests to skip XBL related
tests when XBL is disabled. Also, add an app constant so JS can
check whether XBL is enabled.
Depends on D45614
Differential Revision: https://phabricator.services.mozilla.com/D45615
--HG--
extra : moz-landing-system : lando
This is required due to different backends being used for SVG images and canvas.
Differential Revision: https://phabricator.services.mozilla.com/D44154
--HG--
extra : moz-landing-system : lando
scrollbars-in-landscape-content.html doesn't fail on environments where we don't
use overlay scrollbars because scrollbars for the visual viewport are not
rendered there.
Differential Revision: https://phabricator.services.mozilla.com/D40770
--HG--
extra : moz-landing-system : lando
Converts mozilla.widget.disable-native-theme varcache pref to a static pref and updates uses of its associated global variable with the pref. This also renames the pref to widget.disable-native-theme to group with other widget prefs.
Differential Revision: https://phabricator.services.mozilla.com/D40721
--HG--
extra : moz-landing-system : lando
Due to the sheer number of tests that exhibit a random fuzz with maxDifference=1
and maxDifference=2 with WR on Android, it's easier to just tweak the harness
to autofuzz these away. This adds machinery to do so, and also adds a new
annotation that can be used to disable the autofuzzing on specific tests.
Depends on D36794
Differential Revision: https://phabricator.services.mozilla.com/D36796
--HG--
extra : moz-landing-system : lando
Due to the sheer number of tests that exhibit a random fuzz with maxDifference=1
and maxDifference=2 with WR on Android, it's easier to just tweak the harness
to autofuzz these away. This adds machinery to do so, and also adds a new
annotation that can be used to disable the autofuzzing on specific tests.
Depends on D36794
Differential Revision: https://phabricator.services.mozilla.com/D36796
--HG--
extra : moz-landing-system : lando
Replacing js and text occurences of asyncOpen2
Replacing open2 with open
Differential Revision: https://phabricator.services.mozilla.com/D16885
--HG--
rename : layout/style/test/test_asyncopen2.html => layout/style/test/test_asyncopen.html
extra : moz-landing-system : lando
This duplicates all the reftest tasks except with
dom.serviceWorkers.parent_intercept set to true. For now they are only run on
mozilla-central with linux64/debug.
Depends on D7480
Differential Revision: https://phabricator.services.mozilla.com/D7481
--HG--
extra : moz-landing-system : lando
This allows developers to add "wr-capture" and/or "wr-capture-ref" to a
reftest.list entry and have WR perform a capture after loading the
test and/or ref file for that reftest.
I'm removing this no-longer-used feature because it promotes a behavior
(using the default 0 minimum) that means we never see reports of
unexpected passes when the bugs are fixed, and thus aren't protected
from the bugs regressing again.
Differential Revision: https://phabricator.services.mozilla.com/D3566
--HG--
extra : moz-landing-system : lando
DocShells are associated with outer DOM Windows, rather than Documents, so
having the getter on the document is a bit odd to begin with. But it's also
considerably less convenient, since most of the times when we want a docShell
from JS, we're dealing most directly with a window, and have to detour through
the document to get it.
MozReview-Commit-ID: LUj1H9nG3QL
--HG--
extra : source : fcfb99baa0f0fb60a7c420a712c6ae7c72576871
extra : histedit_source : 5be9b7b29a52a4b8376ee0bdfc5c08b12e3c775a
DocShells are associated with outer DOM Windows, rather than Documents, so
having the getter on the document is a bit odd to begin with. But it's also
considerably less convenient, since most of the times when we want a docShell
from JS, we're dealing most directly with a window, and have to detour through
the document to get it.
MozReview-Commit-ID: LUj1H9nG3QL
--HG--
extra : rebase_source : a13c59d1a5ed000187c7fd8e7339408ad6e2dee6
Kris Maglione
Razvan Maries
Christoph Kerschbaumer
Lee Salzman
Matt Woodrow
Cosmin Sabou
Daosheng Mu
Joel Maher
Bob Owen
Andrew Halberstadt
Stephen A Pohl
Geoff Brown
Alexander Surkov
Jon Coppeard
Jamie Nicol
Brian Grinstead
Brendan Dahl
Paul Zuehlcke
Mike Hommey
Hiroyuki Ikezoe
Kristen Wright
Jonathan Kingston
Kartikaya Gupta
Coroiu Cristina
Henri Sivonen
Andreea Pavel
André Bargull
Noemi Erli
Andrew Swan
L. David Baron
Narcis Beleuzu
