This is also to improve a race condition (bug 1416125), which can happen in the
manifest parsing code on Android jsreftests. For more details, see:
https://bugzilla.mozilla.org/show_bug.cgi?id=1392391#c39
MozReview-Commit-ID: 8qLm1t8agZG
--HG--
extra : rebase_source : 260cbfef9d3bf4be959783f750ca427b34e64486
This replaces reftest's homebrewed chunking algorithm with the one that
all the other test harnesses use in manifestparser.
For now Android will continue to use the reftest based algorithm.
MozReview-Commit-ID: AfUBmQpx3Zz
--HG--
extra : rebase_source : ddc93e7f04e133668a98f050259559a58ff9923c
Instead of parsing the manifests and running the tests all in one go, this will
spawn an extra Firefox instance at the beginning that does nothing but parse the
manifest and dump them to a file.
This will allow the python harness to load and manipulate the test objects, before
sending them back to the JS harness as a list of tests to run. The main motivation
for this change is to implement run-by-manifest, a mode where we restart the
browser in between every test manifest. But there are other benefits as well, like
sharing the chunking logic used by other harnesses and the ability for the python
harness to stuff arbitrary metadata into the test objects.
For now, Android will continue to parse the manifests and run the tests all in one
go. Converting Android to this new mechanism will be left to a follow-up bug.
MozReview-Commit-ID: AfUBmQpx3Zz
--HG--
extra : rebase_source : 82e611e8795150daf01d791ea16517ec1ffb2896
Currently manifest parsing happens within the StartTests method. This method is
already quite large, and this commit series about to make the logic around
gathering tests a lot more complicated.
This commit pulls the manifest parsing out into a new 'ReadTests' method which
is responsible for retrieving the list of tests (however that may be) and then
calling StartTests.
MozReview-Commit-ID: 6ijOqhNaig
--HG--
extra : rebase_source : d494a8665d5019f73631ab8283b6d0c7759ac3bf
This is a simple refactor of manifest.jsm.
We'd like to access the test objects from the parsed manifest in python. This
will allow us implement things like runByManifest (to improve intermittent
stability), share the chunking logic used by other harnesses, and much more.
To do this, we need to JSON serialize all of the test objects and dump them
to a file. The python side can then load the file, make modifications, and
send it back to the JS side to run.
The problem is that we turn the test urls into nsIURI objects as soon as they
are parsed, which isn't JSON serializable. This commit is a simple refactor to
delay this from happening. Instead, we will create the urls in reftest.jsm,
after the modified test objects have been loaded from python. This step will
be implemented by the next commit.
MozReview-Commit-ID: 6ijOqhNaig
--HG--
extra : rebase_source : fbfa259121a8d79bfc44695397d0d1ce08123558
This patch was autogenerated by my decomponents.py
It covers almost every file with the extension js, jsm, html, py,
xhtml, or xul.
It removes blank lines after removed lines, when the removed lines are
preceded by either blank lines or the start of a new block. The "start
of a new block" is defined fairly hackily: either the line starts with
//, ends with */, ends with {, <![CDATA[, """ or '''. The first two
cover comments, the third one covers JS, the fourth covers JS embedded
in XUL, and the final two cover JS embedded in Python. This also
applies if the removed line was the first line of the file.
It covers the pattern matching cases like "var {classes: Cc,
interfaces: Ci, utils: Cu, results: Cr} = Components;". It'll remove
the entire thing if they are all either Ci, Cr, Cc or Cu, or it will
remove the appropriate ones and leave the residue behind. If there's
only one behind, then it will turn it into a normal, non-pattern
matching variable definition. (For instance, "const { classes: Cc,
Constructor: CC, interfaces: Ci, utils: Cu } = Components" becomes
"const CC = Components.Constructor".)
MozReview-Commit-ID: DeSHcClQ7cG
--HG--
extra : rebase_source : d9c41878036c1ef7766ef5e91a7005025bc1d72b
This replaces reftest's homebrewed chunking algorithm with the one that
all the other test harnesses use in manifestparser.
For now Android will continue to use the reftest based algorithm.
MozReview-Commit-ID: AfUBmQpx3Zz
--HG--
extra : rebase_source : cb513d1b3a54ddeb95ce5861d858aad4492de2a6
Instead of parsing the manifests and running the tests all in one go, this will
spawn an extra Firefox instance at the beginning that does nothing but parse the
manifest and dump them to a file.
This will allow the python harness to load and manipulate the test objects, before
sending them back to the JS harness as a list of tests to run. The main motivation
for this change is to implement run-by-manifest, a mode where we restart the
browser in between every test manifest. But there are other benefits as well, like
sharing the chunking logic used by other harnesses and the ability for the python
harness to stuff arbitrary metadata into the test objects.
For now, Android will continue to parse the manifests and run the tests all in one
go. Converting Android to this new mechanism will be left to a follow-up bug.
MozReview-Commit-ID: AfUBmQpx3Zz
--HG--
extra : rebase_source : 955966c07bb650946c7c0e5706856f028335e850
Currently manifest parsing happens within the StartTests method. This method is
already quite large, and this commit series about to make the logic around
gathering tests a lot more complicated.
This commit pulls the manifest parsing out into a new 'ReadTests' method which
is responsible for retrieving the list of tests (however that may be) and then
calling StartTests.
MozReview-Commit-ID: 6ijOqhNaig
--HG--
extra : rebase_source : 16d4e2debcbe95765c4355b9964f62c7e7a417f1
This is a simple refactor of manifest.jsm.
We'd like to access the test objects from the parsed manifest in python. This
will allow us implement things like runByManifest (to improve intermittent
stability), share the chunking logic used by other harnesses, and much more.
To do this, we need to JSON serialize all of the test objects and dump them
to a file. The python side can then load the file, make modifications, and
send it back to the JS side to run.
The problem is that we turn the test urls into nsIURI objects as soon as they
are parsed, which isn't JSON serializable. This commit is a simple refactor to
delay this from happening. Instead, we will create the urls in reftest.jsm,
after the modified test objects have been loaded from python. This step will
be implemented by the next commit.
MozReview-Commit-ID: 6ijOqhNaig
--HG--
extra : rebase_source : 06acb038a4d3e35b3a4158b81b361a9a0ae54337
This is a new issue that gets linted with flake8 3.5.0. Basically you should
never use a blank except: statement.
This will catch all exceptions, including KeyboardInterrupt and SystemExit
(which is likely not intended). If a catch all is needed, use
`except: Exception`. If you *really* mean to also catch KeyboardInterrupt et
al, use `except: BaseException`.
Of course, being specific is often better than a catch all.
MozReview-Commit-ID: FKx80MLO4RN
--HG--
extra : rebase_source : 7c74a7d0d81f2c984b47aff3a0ee3448b791177b
The invalid variable test for #if{,n}def was only checking that the
first character in the variable was alphanumeric or underscore, not
the other characters.
More generally, preprocessor instructions were also cut out such that
whitespaces before and after arguments were part of the arguments.
There's one place in layout/tools/reftest/manifest.jsm that was using
a broken pattern, making the test never true, which, once fixed, unveils
broken tests, so the branch that was never used is removed.
--HG--
extra : rebase_source : d1fe8a299203a29c0906ff99054c326acd135000
This was done using the following script:
37e3803c7a/processors/chromeutils-import.jsm
MozReview-Commit-ID: 1Nc3XDu0wGl
--HG--
extra : source : 12fc4dee861c812fd2bd032c63ef17af61800c70
extra : intermediate-source : 34c999fa006bffe8705cf50c54708aa21a962e62
extra : histedit_source : b2be2c5e5d226e6c347312456a6ae339c1e634b0
This was done using the following script:
37e3803c7a/processors/chromeutils-import.jsm
MozReview-Commit-ID: 1Nc3XDu0wGl
--HG--
extra : source : 12fc4dee861c812fd2bd032c63ef17af61800c70
This was done using the following script:
37e3803c7a/processors/chromeutils-import.jsm
MozReview-Commit-ID: 1Nc3XDu0wGl
--HG--
extra : rebase_source : c004a023389f1f6bf3d2f3efe93c13d423b23ccd
The marionette.defaultPrefs.port and marionette.logging preferences
have been deprecated for some time. We want to move the reftest
harness to use marionette.port and marionette.log.level.
All officially recognised preferences are listed in
testing/marionette/prefs/marionette.js.
MozReview-Commit-ID: H1MHO7Iik4X
--HG--
extra : rebase_source : 883ee2eed272f1a64015cbaaebcfa5dbb45d91bf
Because of the previous patch, the inherited_status is now always
EXPECTED_PASS so we don't need to actually pass it around.
MozReview-Commit-ID: pgMkLgNCOE
--HG--
extra : rebase_source : fce3c66ad7ccaf458befc7c41c9a2a4cdea87875
Without this patch, putting something like
skip-if(Android) include foo/reftest.list
behaves unexpectedly. Instead of skipping the foo/reftest.list file
on Android, it instead "inherits" the skip-if condition down into each
reftest inside foo/reftest.list. If any of those reftests then have a
fuzzy or fuzzy-if(Android,...) annotation, that will override the
skip-if annotation, and the test will still run.
Based on the occurrences of this pattern in the codebase, I believe the
more intuitive semantics of "skip the foo/reftest.list file completely,
on Android" is more desirable. This patch implements the change in
semantics. It also disallows inheriting other statuses down into
included reftest.list files, such as from a fails or fuzzy annotation on
the include statement. These cases don't currently exist in the tree,
but any attempt to do so will now throw an error.
MozReview-Commit-ID: HLJuSYNHvU5
--HG--
extra : rebase_source : d0d31ab63413aeee31d78226e55726836cd6bebf
There are some edge cases in test verification where mozharness requests
reftest verification of a non-reftest. In this case, it is best for the
reftest harness to figure out that the requested file is not a test,
do no work and exit cleanly without complaint.
In order for |mach test| and |mach mochitest| to log an overall summary,
every test harness invocation they make needs to use the same structured
logger (otherwise the affected suite will be missing from the summary).
MozReview-Commit-ID: 8LJw7r8SItk
--HG--
extra : rebase_source : 1417dce3817bae94ad61a5250065c6cbc35857e4
Suite names are currently only used by formatters to print out
an overall summary from |mach test| and |mach mochitest|. So
this doesn't need to be exact and can be tweaked further at a
later date.
If multiple test invocations have the same suite name, their
results will be merged in the overall summary. If a suite name
is missing, the summary will contain a placeholder name.
MozReview-Commit-ID: K1xpb9hUQRX
--HG--
extra : rebase_source : cc8cc8b36255d939dd5dffd3c5444c34951ac8e2
As part of this change, the confusingly named global variable 'state' is
renamed to 'currentURLTargetType', and named "enum" values are assigned
to it rather than raw integers.
MozReview-Commit-ID: FTEOB9wF8Q1
If the reftest harness times out the load of a URL before the 'load' event
has even fired then the error message that we get is 'load failed with unknown
reason'. This isn't very helpful for people unfamiliar with the harness. This
change sets an initial error message that notes that we're waiting on the
'load' event, what the timeout delay is, and what URL we are/were waiting on
loading. This will allow the timeout delay to be compared to log timestamps
and will make it clear whether the timeout occurred for some other URL than
the one we were expecting to load (which would be an error in the harness
logic).
It should now be impossible for the 'load failed with unknown reason' to occur,
but if there is a logic error in the harness code (some race condition?) then
it may still happen.
MozReview-Commit-ID: JOb1kYBpLro
Minor note:
reftests should've turned off uploadEnabled in the first place.
reftests should have unified telemetry on. It's the future.
MozReview-Commit-ID: 9spzuUAXwwP
This requires exposing the reftest chrome package to content, but that should be
OK, and this seems to work...
MozReview-Commit-ID: EWkwqTHW3dg
--HG--
extra : rebase_source : 12cabe4389375ac4c3abd0a9327baf268aab7c1a
Right now, e10s defaults to "on" in Nightly via the default pref
browser.tabs.remote.autostart.2=true. The reftest harness ignores that pref
and attempts to enable it by setting browser.tabs.remote.autostart to true (as
well as disabling the checks for non-mpc-compatible extensions). There's a
bug, however, and we overwrite the value for .autostart by reading
reftest-prefernces.js (which unconditionally sets .autostart to false).
Therefore, the existing code works because it ignores the .autostart.2 pref
and whether we enable e10s is entirely controlled by setting
extensions.e10sBlocksEnabling (the reftest extension is not mpc-compatible).
With this change, we unconditionally set .remote to the value that we want
(and we trust that the only non-mpc-compatible extensions are specially vetted
and part of our build system) so that things work as advertised on the box.
MozReview-Commit-ID: Li5N4NP4PwD
--HG--
extra : rebase_source : 50f842bcf6a13e8daeae8c0c76696c2edeb8f96b
There are two motivations for this change.
First, reftest.jsm has become very large and monolithic. It has lots of global
state and is hard to modify without breaking something. This change is a first
attempt at dividing reftest.jsm into multiple standalone(ish) modules. This
will make it easier to comprehend and extend.
Second, we'd like to implement "run-by-manifest" mode for reftest. This means
we'll restart the browser between each manifest run. This means much of the
state which is normally stored in global variables in reftest.jsm, will instead
need to be stored in python. A prerequisite to doing that is parsing the
manifests before starting the test suite. A prerequisite to that is moving the
manifest parsing code into a standalone JSM. This is the first step.
MozReview-Commit-ID: 42epTs8EU1O
--HG--
extra : rebase_source : 3db27445af0969867857fbfe41add13161a01cb8
(Path is actually r=froydnj.)
Bug 1400459 devirtualized nsIAtom so that it is no longer a subclass of
nsISupports. This means that nsAtom is now a better name for it than nsIAtom.
MozReview-Commit-ID: 91U22X2NydP
--HG--
rename : xpcom/ds/nsIAtom.h => xpcom/ds/nsAtom.h
extra : rebase_source : ac3e904a21b8b48e74534fff964f1623ee937c67
On Windows log lines are buffered in 4kb chunks instead of line buffered. This
means when a log on stdout exceeds this limit, a log from stderr can be
interleaved in the middle. This was causing frequent intermittent failures on
Windows when logging the suite_start message (which is much larger than 4kb).
This patch ensures that instead of redirecting stderr to stdout, we process
that stream independtly (though still using the same output handler). This
means we are guaranteed not to have any log interleaving, but it comes at the
cost of potentially losing the true log ordering. For that reason, this
behaviour is only enabled on Windows. Only the ordering between streams can be
different, and in practice this difference should be really small.
There is currently no good solution for both separating stdout/stderr and
preserving exact log ordering. See bug 798300 for more details.
MozReview-Commit-ID: 5W8I4u15uyM
--HG--
extra : rebase_source : ff6ce214b738dd09081b4b359a49ea7c3b0e4f65
The main bug this fixes is that on reftest, the objdir needs to be added to the
whitelist on Windows. However, this only happens when running on Linux for some
reason.
Changing the --work-path and --obj-path arguments to --sandbox-read-whitelist
was more of a drive-by cleanup than anything necessary.
MozReview-Commit-ID: Dq8ZLETMzeM
--HG--
extra : rebase_source : 3d2cdda125205e76f86235eb373074899fe0789a
If Firefox crashes while mochitests or reftests are running, Marionette
will trigger an IOError exception which currently gets logged immediately,
and causes no post-test checks to be performed. This results in missing
crash and leak checks, and an unclear failure message on Treeherder.
With this change only the IOError from Marionette gets deferred until
all post-test checks are done. This fixes the failure messages, and will
put PROCESS_CRASH or leak log output first.
MozReview-Commit-ID: JCYP5LlPE1m
--HG--
extra : rebase_source : a4a9455402b01db8ef1dbafccc7a726d2927ec03
Currently reftest.jsm uses logger.testEnd(..) to log all kinds of different
potential failures, from proper test failures to harness issues. This means
there are all kinds of edge cases that would cause multiple testEnd messages to
get logged, something the structured log protocol is supposed to prevent.
This modifies the reftest harness to instead use testStatus for everything. The
testEnd call will always be logged with status "OK". This required some changes
to the reftest formatter and reftest selftests.
MozReview-Commit-ID: 8RxsmHW49oy
--HG--
extra : rebase_source : 2f53e63f00bfc5e0e751b0f2bb3dc67477dc07eb
This adds new test verification steps for mochitest/reftest/xpcshell tests
with MOZ_CHAOSMODE=3 (thread scheduling and network thread scheduling).
Enabling all chaos mode features seems to destabilize test verification
so I am only enabling these features for now.
The tests that were run using this UI were removed in bug 652192. (They
did before-patch vs. after-patch comparisons of frame dumps, and didn't
actually have pass/fail state.)
MozReview-Commit-ID: 5qoeaFpEyQg
This installs specialpowers in the test runs so we can trigger crashes/assertions.
MozReview-Commit-ID: 8878OcHv8hU
--HG--
rename : testing/mochitest/tests/python/files/leaks.log => layout/tools/reftest/selftest/files/leaks.log
extra : rebase_source : cee417db30ff76364fa3212a65b2d537c97cc8bd
Complications:
- had to copy ReftestManifest into a test zip
- reftest harness was emitting multiple suite_start log entries with --repeat
- some extra path manipulation required to find reftests
The test plugins - dom/plugins/test/testplugin - are not built on Android.
By assuming that the test plugin is not available on Android, a per-manifest
check and warning can be avoided.
Currently the mochitest and reftest runners are forcing a timeout of 60s
(or 900s for valgrind or debug builds) when calling `start_session` of
Marionette. While this method still offers a timeout parameter, the
timeout should be set via the `startup_timeout` argument for Marionette.
Reason is that Marionette doesn't control the browser instance and
is getting called right after the application gets launched. As such
slow running builds can cause timeouts once it takes longer than 60s
for the Marionette server to get started.
By using `startup_timeout` the timeout will even be configurable via
the command line and mozharness config files.
MozReview-Commit-ID: EV7GklBcJjU
--HG--
extra : rebase_source : 9d3c623c49deb92d68c40ba4410c812c864e06bd
This just adds two basic tests, one for a passing test and another for a
failing one. In mochitest, we use privileged APIs to also tests crashes,
assertions, asan and leaks. But these APIs aren't available to reftests
so I'm not sure how we can test these things.
I figure it's not worth holding the framework up on this though, I'll file
a follow-up to figure out something to do for that.
MozReview-Commit-ID: 59TSbsugT5T
--HG--
extra : rebase_source : 72ecd817017c8b7d55eab879db4f6ad5fecc54c0
Skip tests that are expected to fail in both Stylo and Gecko modes. They would unexpectedly "pass" in styloVsGecko mode when comparing the two failures, which is not a useful result.
MozReview-Commit-ID: 3mOpjU225Q1
--HG--
extra : rebase_source : 22bb5d4e3c5138ef832995eaf5716824f4707ffe
extra : source : d40fb20c9a49d0797c0eeae613a04912b12a28f7
This renames killNamedOrphans to killNamedProc and removes the check for
parent proc id, so that any xpcshell or ssltunnel process is killed before
starting a mochitest or reftest run. For reftests, this is moved out of
the desktop harness and into the remote/android harness, since desktop
reftests do not use xpcshell or ssltunnel.
Since these tests harnesses are already disabling Safe Browsing and tracking
protection, they should also disable the other two features that cause
lists to be downloaded from the Mozilla shavar server.
MozReview-Commit-ID: 2158qRU4XZx
--HG--
extra : rebase_source : 287bfb7580538cca7f58ab035b7141166346c318
The FrameLoaderOwner interface has been implemented in WebIDL for several
years now, so these QIs are simply unnecessary overhead.
MozReview-Commit-ID: LAzvfm5Qhy0
--HG--
extra : rebase_source : 2495c07df21c474f5fabc257ff4db43b0d8047e4
This is straightforward, with only two notable things.
- `#include "nsXPIDLString.h" is replaced with `#include "nsString.h"`
throughout, because all nsXPIDLString.h did was include nsString.h. The
exception is for files which already include nsString.h, in which case the
patch just removes the nsXPIDLString.h inclusion.
- The patch removes the |xpidl_string| gtest, but improves the |voided| test to
cover some of its ground, e.g. testing Adopt(nullptr).
--HG--
extra : rebase_source : 452cc4a08046a1adb1a8099a7e85a1917de5add8
The login reputation checks depend on a server lookup and therefore would
render non-deterministic the performance and correctness of tests.
MozReview-Commit-ID: Bil0rSZsGPT
--HG--
extra : rebase_source : ae06a028c71eee323307ecd4e62bbf1e8a14fe13
The login reputation checks depend on a server lookup and therefore would
render non-deterministic the performance and correctness of tests.
MozReview-Commit-ID: Bil0rSZsGPT
--HG--
extra : rebase_source : c9a641e4bf49c48bf864ed546bf2ae6eb51c27e4
This shouldn't be at module level.
This isn't technically needed for this series. I caught it during
an earlier attempt at teasing out all Python import dependencies
to run `mach`. Why let a good patch go to waste.
MozReview-Commit-ID: FwwdZqcKtpq
--HG--
extra : rebase_source : a8ce2cf452dc8c03371dc384ea4f1be9195b5d30
This mechanically replaces nsILocalFile with nsIFile in
*.js, *.jsm, *.sjs, *.html, *.xul, *.xml, and *.py.
MozReview-Commit-ID: 4ecl3RZhOwC
--HG--
extra : rebase_source : 412880ea27766118c38498d021331a3df6bccc70
This also guards against passing a non-test parameter to reftest.startAfter.
MozReview-Commit-ID: FoqqN4D7sv7
--HG--
extra : rebase_source : 0e5d879467c04803e3ac565b979aceaad3176597
This avoids mistakenly blaming the first reftest in the suite for a crash that
happens during startup. Blaming that reftest can result in fragmentation of the
resulting issues because any bugs filed by sheriffs will be test-specific instead
of grouped together into a single bug.
MozReview-Commit-ID: K4YelNzXxzg
--HG--
extra : rebase_source : b291f365956997bbc3d591addee6817ad3e0ff9b
Test harnesses may use STYLO_FORCE_ENABLED, so we need to check this value when
building the reftest conditions.
MozReview-Commit-ID: 998UMZHNKLl
--HG--
extra : rebase_source : c52e2539c0384c2c300b729ac88fe3381e127509
This adds commands to start a reftest session, run a test, and end the
session. It as assumed that after you start a reftest session you will
just run reftests until you end the session. When starting a session
the user provides a string indicating when screenshots should be
taken, and an object mapping urls to a count of the number of times
that url is expected to be used in the session, to help with
caching. Running the tests takes a url to a test, an expected status,
a timeout, and a nested list of possible references, in which each
entry at a specific level is combined by OR and nested references are
combined by AND.
The implementation is heavilly inspired by the existing reftest
harness, starting a minimal window with no tabs, and loading the urls
directly in there. In order to get a screenshot in the e10s case we
have to pass the DRAW_VIEW and USE_WIDGET_LAYERS flags when taking the
screenshot.
For performance we heavily cache canvases; for references that will be
repeated we cache the full canvas with image, and we also cache a
single canvas to use for all other screenshots to avoid the overhead
of repeatedly creating a new canvas element.
MozReview-Commit-ID: JOFvtmH7tg
This adds commands to start a reftest session, run a test, and end the
session. It as assumed that after you start a reftest session you will
just run reftests until you end the session. When starting a session
the user provides a string indicating when screenshots should be
taken, and an object mapping urls to a count of the number of times
that url is expected to be used in the session, to help with
caching. Running the tests takes a url to a test, an expected status,
a timeout, and a nested list of possible references, in which each
entry at a specific level is combined by OR and nested references are
combined by AND.
The implementation is heavilly inspired by the existing reftest
harness, starting a minimal window with no tabs, and loading the urls
directly in there. In order to get a screenshot in the e10s case we
have to pass the DRAW_VIEW and USE_WIDGET_LAYERS flags when taking the
screenshot.
For performance we heavily cache canvases; for references that will be
repeated we cache the full canvas with image, and we also cache a
single canvas to use for all other screenshots to avoid the overhead
of repeatedly creating a new canvas element.
MozReview-Commit-ID: JOFvtmH7tg
--HG--
extra : rebase_source : ab5a2ef2e450b9bbdc6bc3c9487ed5dfda2c1d4b
This is needed before we can upgrade to flake8 3.3.0, as that version starts flagging these errors.
These files were modified by running:
autopep8 --select E305 --in-place -r <dir>
on the affected directories. I did it one dir at a time and verified the result after each.
MozReview-Commit-ID: FmlsfiKIbtr
--HG--
extra : rebase_source : 9df32258cadff5d27a0e72113c57f782756c0b18
This won't be an issue until the 57 release but at that time, the
reftest extensions will only load if the extensions.legacy.enabled
preference is set to true and Cu.isInAutomation, which depends on
another preference, is true. Set the appropriate prefs here for reftests.
MozReview-Commit-ID: JUWCPTsuJU
--HG--
extra : rebase_source : 042a5696d1926e7ca434a7c4213395176a080549
This catches a common problem where somebody adds a fuzzy annotation on
a test to work around some minor differences. Later the differences go
away, but since the test harness doesn't catch that, nobody is the
wiser. Subsequently a "real" regression can reintroduce differences
which are hidden by the stale fuzzy annotations.
With this patch, if the annotations are set up properly, the test
harness will flag tests as "UNEXPECTED-PASS" when the differences go
away. This will require the patch author to reduce the allowed fuzziness
parameters, and will make it easier to catch subsequent regressions.
MozReview-Commit-ID: B3rGPFLXkCu
This patch extends the "fuzzy" and "fuzzy-if" annotations so that they
accept a range of fuzziness values, much like the "asserts" and
"asserts-if" annotations. If the test produces differences that are
within the specified range, the test is considered as passing. Any
differences outside the ranges are considered failures. If the test is
marked as a "!=" test, then the opposite is true.
If the range given to "fuzzy" or "fuzzy-if" is a single value, it is
interpreted as being a range from zero to the specified value. This is
in contrast to "asserts" and "asserts-if" which match exactly the value
specified. The reason for this is mostly for backwards compatibility
with existing reftest annotation semantics.
MozReview-Commit-ID: 6qUU6FQ5mYP
This simply augments the logging in the reftest harness so that if a reftest
passes due to fuzzy matching, it reports both the actual fuzz numbers and the
maximum allowed fuzz numbers.
MozReview-Commit-ID: G356vBRXYUc
Remove the forbiddenURI pref which was removed in bug 1274893 as well
as browser.safebrowsing.enabled which got renamed in bug 1025965.
Set dummy URLs for all of the network endpoints.
MozReview-Commit-ID: Efk2fv6cC3g
--HG--
extra : rebase_source : 9fbb3eb0fa7f002fe24577a8a0870ec4d1b7cf31
Our linux32-debug build is very slow to startup when running mochitests on aws.
Sometimes we see similar behavior on other linux platforms. Intermittently,
in this environment, startup takes longer than the 120 seconds that marionette
waits, resulting in test failures in bug 1261598. Increasing the marionette
startup timeout to 180 seconds appears to effectively avoid these failures.
Remove suggested and enhanced tiles along with related campaign, frequency-cap, inadjacency, pings, preferences, strings, styles, tests.
MozReview-Commit-ID: FkjaSpSFQHu
--HG--
extra : rebase_source : 1c58ac542180f0abb290639ec1c61b9edf3d0a51
This prefix can't be produced by the standard structured logging
formatters because it doesn't know that the tests are
reftests. Therefore the reftest harness has a hack to add this prefix,
and the unstructured analyzer doesn't work with wpt reftests. This can
simply be solved by making the prefix optional, but looking for the
"INFO - " prefix that mozharness adds, when present, to identify the
start of the TEST- string.
MozReview-Commit-ID: GxWVWOSkAUZ
--HG--
extra : rebase_source : fbd8afa860da0fba4925b6bf2fc715cbf6c2688c
Non-ascii characters in error messages can cause exceptions when processing
structured log messages. The mochitest MessageLogger already handles such
cases; this is a simpler implementation for the reftest OutputHandler.
Previously the harness would check for navigator.plugins[Test Plug-in] in the browser window, but that is now blocked from working. Instead we can simply use an existing getTestPlugin function that enumerates the plugin through nsIPluginHost
MozReview-Commit-ID: Kp48u5iFkSa
Continue to allow non-multiprocessCompatible extensions in automation.
There are a ton of places that would need to be changed, many of which
will be changing soon anyway with the non-webextensions change in 57
so this is mostly the expedient route to keeping the tree green.
MozReview-Commit-ID: EZZoDVdhLfy
--HG--
extra : rebase_source : f83472bc1c88dd0deadbe485d9002499027ff07f
Continue to allow non-multiprocessCompatible extensions in automation.
There are a ton of places that would need to be changed, many of which
will be changing soon anyway with the non-webextensions change in 57
so this is mostly the expedient route to keeping the tree green.
MozReview-Commit-ID: EZZoDVdhLfy
--HG--
extra : rebase_source : 34aa762917566b052ade6372280caed72fbfbe9a
This patch does a few things:
a) Adds the resources location from the .app directory to the read whitelist
b) When it's a non-packaged build, mach run (and various mach tests) set an environment variable for the repo location which we allow reads from.
r=haik,froydnj
MozReview-Commit-ID: KNvAoUs5Ati
--HG--
extra : rebase_source : 81ba8bfee0ca96979cf8e30d75cdd47f06bc10ea
This patch does a few things:
a) Adds the resources location from the .app directory to the read whitelist
b) When it's a non-packaged build, mach run (and various mach tests) set an environment variable for the repo location which we allow reads from.
r=haik,froydnj
MozReview-Commit-ID: KNvAoUs5Ati
--HG--
extra : rebase_source : f637acff32fc8582732de932503dd696abc57877
This eliminates a 2 minute timeout seen at the end of Android mochitests
and reftests. Attempts to shutdown the web server were failing because
they were directed at IP 10.0.2.2 -- the loopback address for the
Android emulator.
The devicemanager killProcess() is updated to use force-stop first, then
use kill if force-stop does not work.
Browser test harnesses are updated to check if killProcess() worked, and
warn if it failed.
Pending crash reports are stored in UAppData, which is typically
outside of the test profile. Pending crash reports left by one test
may affect future tests, so it is important that they be deleted
between tests. However, since UAppData crash data may be important
to developers running tests outside of automation, the harness cannot
delete them unless requested with the --cleanup-crashes option.
The recommended Marionette preferences for use in automation are not so
useful when testing the browser itself. Many of the tests for Firefox
have highly specific requirements, and setting the recommended Marionette
preferences could for these mean a deterioration of test coverage.
The majority of test harnesses utilising Marionette use it to install
unsigned add-ons at runtime, and these preferences are not relevant in
this context.
To avoid an unfortunate and unintended regression in test coverage,
we skip using the recommended Marionette preferences when we test the
browser itself.
MozReview-Commit-ID: 4DeHZfQEX7d
--HG--
extra : rebase_source : bf31d336df7c4c28c04454338a912632087aa3ff
reftest cannot use testing-common, so we should include AsyncSpellCheckTestHelper.jsm in reftest.jar.
Previous fix has typo of chrome vs resource. It should be chrome://.
MozReview-Commit-ID: KeyDLjc9AUI
--HG--
extra : rebase_source : 5bf2e6f4105f3437fb3c88410a246e1b85b1bf1d
CLOSED TREE
Backed out changeset 941e0f9ff9a7 (bug 1351074)
Backed out changeset 4fdf3b87a70b (bug 1351074)
Backed out changeset 586428f69838 (bug 1351074)
reftest cannot use testing-common, so we should include AsyncSpellCheckTestHelper.jsm in reftest.jar.
MozReview-Commit-ID: KeyDLjc9AUI
--HG--
extra : rebase_source : 0c967b38faf506c2e93abc8707e27cfa5388de00
sutagent is no longer built or used; devicemanagerSUT is completely
unused. After this change, devicemanagerADB is the only implementation of
devicemanager, and test harness options like --dm_trans are eliminated.
sutagent is no longer built or usedr; devicemanagerSUT is completely
unused. After this change, devicemanagerADB is the only implementation of
devicemanager, and the --dmTrans and similar options have been removed
from test harnesses and mach commands.
This makes two changes:
* adds "on the line" to clarify what last means
* adds "when combining <failure-type> from the manifest include and the
test line" to clarify that the parenthetical only applies to combining
at different levels, and not within a line
DONTBUILD
--HG--
extra : rebase_source : 4e45753f11b20313ed010ec8d01e0403b89591fd
extra : amend_source : c1f4acb341f0cb2f713080e73c686a5e67aed521
This makes the code work the way it looks like it was intended to work
based on what was already in output.py.
MozReview-Commit-ID: 8LJHxlfiQQ8
--HG--
extra : rebase_source : 46b2435ddbd3e8485536806d9fc7987599227f16
Prior to this patch, dumping of style contexts in the layout
debugger would only dump the subtree associated with the root
frame, which just covers the viewport, canvas, and associated
scrollbars.
I think this has been broken since bug 236921 (2004) and
bug 378693 (2007) properly disconnected the root element from the
styles for the viewport and canvas.
This patch just manually dumps the root frame and the root
element's frame, since the style set no longer maintains a list
of all of the style context roots.
MozReview-Commit-ID: 5LEnbNtngN2
--HG--
extra : transplant_source : v%BB%09%C8M8%FE%25%A9%A6%3D%EA%BD%BE%DF7%08%9A%85m
With this pref turned on, channels with the LOAD_CLASSIFY_URI load flag
will return true from the nsIHttpChannel::IsTrackingResource() API after
being loaded if their URL appears on the TP list.
Andrew Halberstadt
Gurzau Raul
Andrew McCreight
Sebastian Hengst
Florian Quèze
Mike Hommey
Boris Zbarsky
Geoff Brown
Kris Maglione
Cosmin Sabou
Brindusan Cristian
Andreas Tolfsen
Kartikaya Gupta
Sylvestre Ledru
Dorel Luca
Xidorn Quan
Narcis Beleuzu
Jonathan Watt
Emilio Cobos Álvarez
Noemi Erli
Jan de Mooij
Ed Lee
Chris H-C
Phil Ringnalda
Henrik Skupin
Masatoshi Kimura
Matt Woodrow ext:(%2C%20Miko%20Mynttinen%20%3Cmikokm%40gmail.com%3E%2C%20Timothy%20Nikkel%20%3Ctnikkel%40gmail.com%3E)
Blake Kaplan
Alex Gaynor
Tobias Schneider
Nicholas Nethercote
Nathan Froyd
Marina Samuel
Henri Sivonen
L. David Baron
Chris Peterson
Gian-Carlo Pascutto
J. Ryan Stinnett
Francois Marier
Wes Kocher
Gregory Szorc
Bob Owen
k88hudson
Carsten "Tomcat" Book
Dan Mosedale ext:(%3E)
Samael Wang
Jonathan Kew
Haik Aftandilian
Thomas Nguyen
James Graham
David Anderson
Olli Pettay
Dão Gottwald
Andrew Swan
Shing Lyu
Mike Cooper
Bobby Holley
Benjamin Smedberg
Andreas Tolfsen
Makoto Kato
Tooru Fujisawa
Mason Chang
Andrea Marchesini
Deepa
Ryan Hunt
Jet Villegas
Mark Banner
Ehsan Akhgari
Morris Tseng