зеркало из https://github.com/mozilla/gecko-dev.git
a9dd0b2e3e
Currently the RefTest manifest parser has 5 possible statuses:
0 => EXPECTED_PASS
1 => EXPECTED_FAIL
2 => EXPECTED_RANDOM
3 => EXPECTED_DEATH (aka skip)
4 => EXPECTED_FUZZY
In the manifests, the last status annotation that appears on the line will take
precedence. For example:
skip-if(true) fails-if(true) == test1.html ref.html
fails-if(true) skip-if(true) == test2.html ref.html
The first test will have an expected status equal to EXPECTED_FAIL, whereas the
second one will be EXPECTED_DEATH. The same holds true for any combination of
'fail/random/skip/fuzzy' annotations. This means developers need to be very
careful about the order they specify these annotations as getting the order
wrong can easily lead to unexpected behaviour.
With the introduction of defaults in bug 1616368, the risk of unexpected behaviour
is far greater. Since defaults are simply prepended to the test line, a manifest
that looks like:
defaults skip-if(true)
== test1.html ref.html
fails-if(true) == test2.html ref.html
will actually skip the first test, but run the second (since the fails-if
overwrites EXPECTED_DEATH with EXPECTED_FAIL).
The root of the problem appears to be that 'skip' and 'fuzzy' are not actually
test statuses. They are modifiers that affect how we run the test, but don't
actually affect whether the test is expected to pass or fail.
Therefore, this patch solves the problem by making 'skip/skip-if' its own thing
that does not get overwritten by other failure types. In otherwords, a 'skip-if'
can appear before or after a 'fails-if' and it will have the same meaning.
Differential Revision: https://phabricator.services.mozilla.com/D64457
--HG--
extra : moz-landing-system : lando
|
||
|---|---|---|
| .. | ||
| layout-debug | ||
| recording | ||
| reftest | ||
| tests | ||