Increases WPT wdspec timeouts to more realistic values. Because wdspec
tests interact with the browser from an OOP program, they require more
time to run. Interactive browser tests are also known for generally
being more expensive to run.
25 seconds for the default timeout and 120 seconds for the long timeout
are values picked out of the air and likely needs to be further refined
in the future. It is however the current belief that this moves us in
the right direction.
Further improvements to this approach may involve letting wdspec tests
define timeouts on a per-file or a per-test function level through the
use of pytest-timeouts, but this is purely speculative at this point.
It is the current recommendation to adjust the number of tests and the
runtime duration of the tests in a file according to these new defaults.
MozReview-Commit-ID: 4I3Xz9G6lzv
--HG--
extra : rebase_source : 5ec7439e736dc9978828e420bd31195e63130fed
Certain test types have a need for other defaults than the
wpttest.DEFAULT_TIMEOUT and wpttest.LONG_TIMEOUT values. This patch
changes wptrunner to define default- and long timeouts on a test type
level. This allows a test type to override the default durations defined
in the abstract Test.default_timeout and Test.long_timeout.
Concrete classes, such as ReftestTest and WdspecTest, may override these
class properties.
MozReview-Commit-ID: IS6df5vuIDC
--HG--
extra : rebase_source : a3f37d4524902f2b0d54e14126b57da327f0ec06
Implement gecko leak checking for web-platform-tests, but do not
enable it yet due to oranges. Allows disabling leak checks for a
specific test using a 'leaks' key in test metadata. This causes the
browser to restart before running the (group of) tests with leak
checking disabled. The feature can be enabled by passing --leak-check
on the command line.
MozReview-Commit-ID: 1kBnJkOaeu8
--HG--
extra : rebase_source : 2ad24889743a9a76a5027d68c46223617abd0f72
If getting a test from the queue failed, we would loop forever rather
than trying the next queue.
MozReview-Commit-ID: K5YxCZPtx4l
--HG--
extra : rebase_source : f30618d0b26b740ba9748b912507fd03558f6d47
For gecko builds with --disable-optimize everything is very very slow;
enough that the fixed 60s timeout for the browser to start is
insufficient, and various marionette timeouts may also be hit. To
alleviate this problem either disable timeouts or multiply them by the
timeout multiplier which is generally set to around 3 for debug
builds. This seems sufficient to solve the problem on the hardware
developers actually have.
MozReview-Commit-ID: I3zHJGXlpnd
--HG--
extra : rebase_source : 78b14a7b48b8ddfd0ad249a84b8cc94872639cfd
This change renames the following Marionette preferences:
marionette.defaultPrefs.enabled marionette.enabled
marionette.defaultPrefs.port marionette.port
marionette.force-local marionette.forcelocal
marionette.logging marionette.log.level
The old preference names are still usable, but are henceforth considered
deprecated. They will be removed when Firefox 55 ships.
It also registers these preferences in Firefox so that they are
discoverable through about:config.
This patch also refactors testing/marionette/components/marionette.js.
MozReview-Commit-ID: 1dAMQS2e0og
--HG--
extra : rebase_source : e59a9e656442d8e8cebeaf5e741dc8306436bc22
In order to support leak checking in web-platform-tests, we have to
support restarting the browser before a test starts, which is
complicated by the architecture of wptrunner that puts the TestRunner
and associated Executor in a separate process. In the previous
architecture the Executor process grabbed the test to run from the
queue so it wasn't possible to restart the browser once the test was
known without considerable difficulty.
The primary goal of this refactor is to give the TestRunner access to
the test before it is passed to the executor. However imlementing this
was complicated by the poor structure of the existing code. To make
things more comprehensible the TestRunner class is factored into a
more explicit state machine, with a run() loop that either transitions
the runner into the next state or waits for an external command which
causes such a state transition.
MozReview-Commit-ID: 7cvSkYlCSKe
--HG--
extra : rebase_source : 47eaeb69f4cf97d7b560dac301585a7df77475fc
This avoids a cache collision between directories in the main test set
and vendor-specific tests under some different metadata root.
MozReview-Commit-ID: 5WaEtaLJkVD
Ensure that the webdriver tests can access the web-platform.test
hostnames in the same way as other tests by setting the necessary
Firefox pref as part of the capabilities.
MozReview-Commit-ID: At5ENJY7UXA
This avoids a cache collision between directories in the main test set
and vendor-specific tests under some different metadata root.
MozReview-Commit-ID: 5WaEtaLJkVD
Ensure that the webdriver tests can access the web-platform.test
hostnames in the same way as other tests by setting the necessary
Firefox pref as part of the capabilities.
MozReview-Commit-ID: At5ENJY7UXA
I recently changed it to function scoped, which was a mistake.
pytest complains if it is not module scoped.
MozReview-Commit-ID: GYP9Ky1avks
--HG--
extra : rebase_source : 0776803493f7175d1c408c50f06fa6112c196abc
It is required to skip certain WebDriver tests imperatively at runtime,
such as for the Get Window Size and Get Window Position commands, which
are only available at certain configurations.
MozReview-Commit-ID: 9ERYkYm34iI
--HG--
extra : rebase_source : 5fed98284ab6f32009cdb1fa819cf2337adb4a73
I recently changed it to function scoped, which was a mistake.
pytest complains if it is not module scoped.
MozReview-Commit-ID: GYP9Ky1avks
--HG--
extra : rebase_source : ad96a48fe19c987f3021225f11875d6e80858ccf
It is required to skip certain WebDriver tests imperatively at runtime,
such as for the Get Window Size and Get Window Position commands, which
are only available at certain configurations.
MozReview-Commit-ID: 9ERYkYm34iI
--HG--
extra : rebase_source : e30e2fa1473e25511dcec0361ad4a7eecce04e67
Before this, the default of "equal_time" resulted in W7 jobs on Windows
debug builds executing over 6 times slower than the shortest chunk.
Chunking by hash of test directory makes the chunk sizes more consistent.
They aren't as consistent as a full path hashing or random chunking. But
it is an improvement.
MozReview-Commit-ID: 7bUSxY8GhH7
--HG--
extra : rebase_source : daa42b3e757a18fef1233338c0686154b387851d
extra : source : c4ae7809108445a184168a62dac6dab894c7f292
Per discussion in the bug, we want automation to behave like local
development mode with regards to running tests in a directory. That
means we want all tests in a directory to be executed together.
We introduce a directory hash chunker. It is like the HashChunker
except it hashes the directory of the test (not the full path). This
ensures that all tests in the same directory end up in the same
chunk.
Compared to HashChunker, this will likely make slow directories
contribute to higher variance in chunk execution times.
MozReview-Commit-ID: CeV0Gi6NRRp
--HG--
extra : rebase_source : 88279fbad4a2f0ae3fd6d27621ecc0d3c7e6d8ca
The built-in hash() function uses the backing memory address for
hashing. This is essentially random. Switch to md5 so input is
consistently hashed across processes.
MozReview-Commit-ID: D52uzttE5hc
--HG--
extra : rebase_source : b7f15e519ec0ba05d0a310fe0470d82893a0a07c
extra : source : 4c8e1c96a27df961c1dd2e984745d06757e8a2d4
./mach web-platform-tests tries to access the tests at
http://web-platform.test/. If an HTTP proxy is configured, it tries to
access it via the proxy, which fails. Disabling the proxy entirely
fixes the problem and should probably have no adverse effects, although
a cleaner solution would be to skip proxies only for that domain (as is
done with localhost).
MozReview-Commit-ID: IkIRLwbH16P
Andreas Tolfsen
James Graham
Sebastian Hengst
William Lachance
Shawn Huang
Wes Kocher
Maja Frydrychowicz
Thomas Charles
Ryan VanderMeulen
Gregory Szorc
Aryeh Gregor