This patch adopts Logger#managerLevelFromPref from Log.jsm to set
and keep track of the Marionette logger's verbosity.
This has the advantage that we do not have to roll separate
implementations of Log for the child- and parent processes. It also
has the upside that the log level will be reflected when changed
at runtime through the use of an observer.
Reftest manifest parsing fails to recognize individual tests with ? in the name. We missed this as in aou
Tes Plan:
Differential Revision: https://phabricator.services.mozilla.com/D1667
This moves reftest-preferences.js to:
testing/profiles/reftest/user.js
Developers can also now add extensions to:
testing/profiles/reftest/extensions
MozReview-Commit-ID: IVLsT5MWtcJ
--HG--
rename : layout/tools/reftest/reftest-preferences.js => testing/profiles/reftest/user.js
extra : rebase_source : 1725627d439998d92545e0d4693fbb76a266945f
This happens when an nsIFrame* that builds an nsDisplayWrapList is deleted, but then the memory is immediately reused for another frame that builds the same type display item, within the same display list.
PreProcessDisplayLists chooses not to descend into the nsDisplayWrapList for the deleted frame, and so mOldItems remains uninitialized for the old sublist.
When adding the new instance, IsChanged returns false, since the pointers are the same, and we're checking HasDeletedFrame on the new instance (where it's never true), instead of the old. We then recurse into MergeDisplayLists, with an uninitialized mOldItems array, and crash.
I haven't added a test because I haven't yet figured out how to create a minimal testcase, and the test would rely on implementation details of the frame allocator to remain unchanged to be useful.
MozReview-Commit-ID: pHimEvfAND
--HG--
extra : rebase_source : 10baa8cabf1eca8d592f2be0eb7bc7c9461f9785
Run-by-manifest is a mode where we restart Firefox between every manifest of
tests. It sacrifices a little bit of runtime for better test isolation and
improved stability.
This turns run-by-manifest on for all platforms except Android. It also skips
jsreftests and crashtests for now (mostly to limit the scope of what was
landing all at once). Follow-ups will be filed to get it turned on in those
places.
MozReview-Commit-ID: DmvozAIPE5Q
--HG--
extra : rebase_source : 67470894a7aa0b3189380a4874495395401909bb
The point of start-after-crash was to resume running tests after a crash so we
don't miss out on test coverage when a crash happens. Now that we're close to
landing run-by-manifest, this feature is as necessary since only tests that
appear later in the same manifest as the crashing test will be skipped.
Another reason to remove it, is that it's current implementation is buggy. It
assumes tests have a unique ID (they don't), which means we could accidentally
get into an infinite loop if the wrong test happens to crash at the wrong time.
A third reason to remove it is that it adds a lot of complexity to the harness.
Especially now with run-by-manifest, supporting both would require handling a
lot of different edge cases and make things much more complicated than the
already are.
All that being said, it would still provide value. Not only would it allow us
to resume tests in the same manifest, but run-by-manifest isn't yet enabled on
Android, jsreftest or crashtest (though we hope to get it enabled in those
places in the future). So it may be worth re-implementing start-after-crash
again based on the new manifest parsing mechanism that run-by-manifest uses. It
should be possible to implement it a lot more cleanly now. This work will have
to be left to a follow-up.
MozReview-Commit-ID: P2hh5VecKW
--HG--
extra : rebase_source : 809d5103b8d513c709aa950b102d3efee28fb205
This implements a chunk_by_manifest algorithm. It is similar to chunk_by_slice
in that it tries to make an even number of tests run in each chunk. However,
unlike chunk_by_slice it will guarantee that tests in the same manifest will
all run in the same chunk. This makes it suitable to use with run-by-manifest.
This means the chunks won't be perfect (as manifests are differnet sizes). It
is also prone to more randomization, similar to chunk-by-runtime.
In fact, this algorithm is nearly identical to the chunk-by-runtime one, so it
was refactored out to a base class.
MozReview-Commit-ID: HI2ByxW0i8V
--HG--
extra : rebase_source : e066c034b85222d26bafe6873a80366d5bd9df9e
This replaces reftest's homebrewed chunking algorithm with the one that
all the other test harnesses use in manifestparser.
For now Android will continue to use the reftest based algorithm.
MozReview-Commit-ID: AfUBmQpx3Zz
--HG--
extra : rebase_source : ddc93e7f04e133668a98f050259559a58ff9923c
Instead of parsing the manifests and running the tests all in one go, this will
spawn an extra Firefox instance at the beginning that does nothing but parse the
manifest and dump them to a file.
This will allow the python harness to load and manipulate the test objects, before
sending them back to the JS harness as a list of tests to run. The main motivation
for this change is to implement run-by-manifest, a mode where we restart the
browser in between every test manifest. But there are other benefits as well, like
sharing the chunking logic used by other harnesses and the ability for the python
harness to stuff arbitrary metadata into the test objects.
For now, Android will continue to parse the manifests and run the tests all in one
go. Converting Android to this new mechanism will be left to a follow-up bug.
MozReview-Commit-ID: AfUBmQpx3Zz
--HG--
extra : rebase_source : 82e611e8795150daf01d791ea16517ec1ffb2896
This replaces reftest's homebrewed chunking algorithm with the one that
all the other test harnesses use in manifestparser.
For now Android will continue to use the reftest based algorithm.
MozReview-Commit-ID: AfUBmQpx3Zz
--HG--
extra : rebase_source : cb513d1b3a54ddeb95ce5861d858aad4492de2a6
Instead of parsing the manifests and running the tests all in one go, this will
spawn an extra Firefox instance at the beginning that does nothing but parse the
manifest and dump them to a file.
This will allow the python harness to load and manipulate the test objects, before
sending them back to the JS harness as a list of tests to run. The main motivation
for this change is to implement run-by-manifest, a mode where we restart the
browser in between every test manifest. But there are other benefits as well, like
sharing the chunking logic used by other harnesses and the ability for the python
harness to stuff arbitrary metadata into the test objects.
For now, Android will continue to parse the manifests and run the tests all in one
go. Converting Android to this new mechanism will be left to a follow-up bug.
MozReview-Commit-ID: AfUBmQpx3Zz
--HG--
extra : rebase_source : 955966c07bb650946c7c0e5706856f028335e850
This is a new issue that gets linted with flake8 3.5.0. Basically you should
never use a blank except: statement.
This will catch all exceptions, including KeyboardInterrupt and SystemExit
(which is likely not intended). If a catch all is needed, use
`except: Exception`. If you *really* mean to also catch KeyboardInterrupt et
al, use `except: BaseException`.
Of course, being specific is often better than a catch all.
MozReview-Commit-ID: FKx80MLO4RN
--HG--
extra : rebase_source : 7c74a7d0d81f2c984b47aff3a0ee3448b791177b
The marionette.defaultPrefs.port and marionette.logging preferences
have been deprecated for some time. We want to move the reftest
harness to use marionette.port and marionette.log.level.
All officially recognised preferences are listed in
testing/marionette/prefs/marionette.js.
MozReview-Commit-ID: H1MHO7Iik4X
--HG--
extra : rebase_source : 883ee2eed272f1a64015cbaaebcfa5dbb45d91bf
There are some edge cases in test verification where mozharness requests
reftest verification of a non-reftest. In this case, it is best for the
reftest harness to figure out that the requested file is not a test,
do no work and exit cleanly without complaint.
In order for |mach test| and |mach mochitest| to log an overall summary,
every test harness invocation they make needs to use the same structured
logger (otherwise the affected suite will be missing from the summary).
MozReview-Commit-ID: 8LJw7r8SItk
--HG--
extra : rebase_source : 1417dce3817bae94ad61a5250065c6cbc35857e4
Suite names are currently only used by formatters to print out
an overall summary from |mach test| and |mach mochitest|. So
this doesn't need to be exact and can be tweaked further at a
later date.
If multiple test invocations have the same suite name, their
results will be merged in the overall summary. If a suite name
is missing, the summary will contain a placeholder name.
MozReview-Commit-ID: K1xpb9hUQRX
--HG--
extra : rebase_source : cc8cc8b36255d939dd5dffd3c5444c34951ac8e2
Right now, e10s defaults to "on" in Nightly via the default pref
browser.tabs.remote.autostart.2=true. The reftest harness ignores that pref
and attempts to enable it by setting browser.tabs.remote.autostart to true (as
well as disabling the checks for non-mpc-compatible extensions). There's a
bug, however, and we overwrite the value for .autostart by reading
reftest-prefernces.js (which unconditionally sets .autostart to false).
Therefore, the existing code works because it ignores the .autostart.2 pref
and whether we enable e10s is entirely controlled by setting
extensions.e10sBlocksEnabling (the reftest extension is not mpc-compatible).
With this change, we unconditionally set .remote to the value that we want
(and we trust that the only non-mpc-compatible extensions are specially vetted
and part of our build system) so that things work as advertised on the box.
MozReview-Commit-ID: Li5N4NP4PwD
--HG--
extra : rebase_source : 50f842bcf6a13e8daeae8c0c76696c2edeb8f96b
On Windows log lines are buffered in 4kb chunks instead of line buffered. This
means when a log on stdout exceeds this limit, a log from stderr can be
interleaved in the middle. This was causing frequent intermittent failures on
Windows when logging the suite_start message (which is much larger than 4kb).
This patch ensures that instead of redirecting stderr to stdout, we process
that stream independtly (though still using the same output handler). This
means we are guaranteed not to have any log interleaving, but it comes at the
cost of potentially losing the true log ordering. For that reason, this
behaviour is only enabled on Windows. Only the ordering between streams can be
different, and in practice this difference should be really small.
There is currently no good solution for both separating stdout/stderr and
preserving exact log ordering. See bug 798300 for more details.
MozReview-Commit-ID: 5W8I4u15uyM
--HG--
extra : rebase_source : ff6ce214b738dd09081b4b359a49ea7c3b0e4f65
The main bug this fixes is that on reftest, the objdir needs to be added to the
whitelist on Windows. However, this only happens when running on Linux for some
reason.
Changing the --work-path and --obj-path arguments to --sandbox-read-whitelist
was more of a drive-by cleanup than anything necessary.
MozReview-Commit-ID: Dq8ZLETMzeM
--HG--
extra : rebase_source : 3d2cdda125205e76f86235eb373074899fe0789a
If Firefox crashes while mochitests or reftests are running, Marionette
will trigger an IOError exception which currently gets logged immediately,
and causes no post-test checks to be performed. This results in missing
crash and leak checks, and an unclear failure message on Treeherder.
With this change only the IOError from Marionette gets deferred until
all post-test checks are done. This fixes the failure messages, and will
put PROCESS_CRASH or leak log output first.
MozReview-Commit-ID: JCYP5LlPE1m
--HG--
extra : rebase_source : a4a9455402b01db8ef1dbafccc7a726d2927ec03
This adds new test verification steps for mochitest/reftest/xpcshell tests
with MOZ_CHAOSMODE=3 (thread scheduling and network thread scheduling).
Enabling all chaos mode features seems to destabilize test verification
so I am only enabling these features for now.
Currently the mochitest and reftest runners are forcing a timeout of 60s
(or 900s for valgrind or debug builds) when calling `start_session` of
Marionette. While this method still offers a timeout parameter, the
timeout should be set via the `startup_timeout` argument for Marionette.
Reason is that Marionette doesn't control the browser instance and
is getting called right after the application gets launched. As such
slow running builds can cause timeouts once it takes longer than 60s
for the Marionette server to get started.
By using `startup_timeout` the timeout will even be configurable via
the command line and mozharness config files.
MozReview-Commit-ID: EV7GklBcJjU
--HG--
extra : rebase_source : 9d3c623c49deb92d68c40ba4410c812c864e06bd
This just adds two basic tests, one for a passing test and another for a
failing one. In mochitest, we use privileged APIs to also tests crashes,
assertions, asan and leaks. But these APIs aren't available to reftests
so I'm not sure how we can test these things.
I figure it's not worth holding the framework up on this though, I'll file
a follow-up to figure out something to do for that.
MozReview-Commit-ID: 59TSbsugT5T
--HG--
extra : rebase_source : 72ecd817017c8b7d55eab879db4f6ad5fecc54c0
This renames killNamedOrphans to killNamedProc and removes the check for
parent proc id, so that any xpcshell or ssltunnel process is killed before
starting a mochitest or reftest run. For reftests, this is moved out of
the desktop harness and into the remote/android harness, since desktop
reftests do not use xpcshell or ssltunnel.
This also guards against passing a non-test parameter to reftest.startAfter.
MozReview-Commit-ID: FoqqN4D7sv7
--HG--
extra : rebase_source : 0e5d879467c04803e3ac565b979aceaad3176597
Andreas Tolfsen
Chris AtLee
Bogdan Tara
Joel Maher
Andrew Halberstadt
Matt Woodrow
Geoff Brown
Gurzau Raul
Cosmin Sabou
Sebastian Hengst
Dorel Luca
Henrik Skupin
Blake Kaplan
Alex Gaynor
Gian-Carlo Pascutto
Bob Owen
Jonathan Kew
Kartikaya Gupta
Haik Aftandilian