This moves the Rust crate mozrunner into central from GitHub.
The old repository will be graveyarded:
https://github.com/jgraham/rust_mozrunner
The git history is not considered important, hence this does not
overlay that onto central like we did for testing/geckodriver and
testing/webdriver.
MozReview-Commit-ID: J4ZYdow2Lkw
--HG--
extra : rebase_source : 1b499b708105a89a5fa3ae6ecac71c4946e20755
This marks **/docs/** as exclusively docs, and code that is autodoc'd as
inclusively docs.
That means that a change that purely modifies documentation files will *only*
run `docs` tasks, while a change that modifies autodoc'd source code will
*additionaly* run `docs` tasks. The tasks do not run by default.
MozReview-Commit-ID: G9tOK0AwtrI
--HG--
extra : rebase_source : 8dd971e5c9b0eb5f47895664a4ea49442f303ecb
extra : source : 0881de9b2b5e36ec37cc866f1d4af109da57a919
This enables the syntax like:
./mach try fuzzy dom/indexedDB
This will open up the fzf interface like normal, except only tasks
that have tests under dom/indexedDB will be selectable (and there
will only be one chunk per configuration).
This can be combined with -q/--query like normal:
./mach try fuzzy dom/indexedDB -q "!pgo !cov !asan"
When the tasks get scheduled, only the tests under the specified
path(s) will run within the harness.
MozReview-Commit-ID: IHRXXi5mB4G
--HG--
extra : rebase_source : 8a89f255591e6dfa31b1420196c4698f2015d10c
This moves the Rust crate mozversion into central from GitHub.
The old repository will be graveyarded:
https://github.com/jgraham/mozversion
The git history is not considered important, hence this does not
overlay that onto central like we did for testing/geckodriver and
testing/webdriver.
MozReview-Commit-ID: HeBggGmGsg6
--HG--
extra : rebase_source : 14f6943394bd7b6e8daa7a35b29bc209b7ac9ad4
This removes the subcommands for "./mach geckodriver", reverting
it back to have the meaning of running the geckodriver binary.
The build- and test commands are now integrated with mach, which
means you can run "./mach build testing/geckodriver" and "./mach
test testing/geckodriver" to run tests. This is backed by a new
top-level "./mach geckodriver-test" command, which we will not be
announcing.
MozReview-Commit-ID: CiQsfNqrvIp
--HG--
extra : rebase_source : 6c492b7e1128e4858e42ae4bb35ab4b29564dbeb
This makes several changes to make the 'mach' format cleaner and easier to
read. Some of the changes include:
* No longer print the 'action' no matter what. Printing the action for things
like 'log' or 'process_output' was redundant and caused verbosity. Now this
is done on a case by case basis (things like TEST-START/TEST-END will still
have their actions printed).
* Color coded the process id for 'process_output' actions. This is a dim cyan
to avoid conflicts with other actions.
* No longer quoting 'process_output' messages
* No longer printing thread information. In 99% of the case, this was just
dumping 'MainThread' over and over again. Perhaps printing this could be an
option on the formatter.
* Muted timestamps to help the important parts stand out better
* Colorized suite summary headings
* Unexpected statuses in _format_expected() are always red (never yellow).
This is to help make it stand out from all the other yellow text that gets
printed.
* Internal cleanup/refactoring
MozReview-Commit-ID: LAuYfqYkUPe
--HG--
extra : rebase_source : 6cab1bc3e38838f200f90acc2fff8dcad3d394f3
This makes several changes to make the 'mach' format cleaner and easier to
read. Some of the changes include:
* No longer print the 'action' no matter what. Printing the action for things
like 'log' or 'process_output' was redundant and caused verbosity. Now this
is done on a case by case basis (things like TEST-START/TEST-END will still
have their actions printed).
* Color coded the process id for 'process_output' actions. This is a dim cyan
to avoid conflicts with other actions.
* No longer quoting 'process_output' messages
* No longer printing thread information. In 99% of the case, this was just
dumping 'MainThread' over and over again. Perhaps printing this could be an
option on the formatter.
* Muted timestamps to help the important parts stand out better
* Colorized suite summary headings
* Unexpected statuses in _format_expected() are always red (never yellow).
This is to help make it stand out from all the other yellow text that gets
printed.
* Internal cleanup/refactoring
MozReview-Commit-ID: LAuYfqYkUPe
--HG--
extra : rebase_source : 41aa8651fc8d182bfcbd57c1d97b1bee437d478c
When 'summary_on_shutdown' is True (which is the case for |mach test| and
|mach mochitest|, the tbplformatter will now print an overall summary at
the end of the log run.
MozReview-Commit-ID: 9ieqJRcON8e
--HG--
extra : rebase_source : a27f6230c4d2daaa547e6fede24ba0c9ef55bfc0
When 'summary_on_shutdown' is True (which is the case for |mach test| and |mach
mochitest|), BaseSummaryFormatters will save the summary information until the
'shutdown' action is received at the end of the logger's lifetime.
Summary information will no longer be dumped on 'suite_end'.
MozReview-Commit-ID: HKtVr5PxfOy
--HG--
extra : rebase_source : f350f09111deb510b27a4e55797243dda3160869
The mach formatter gathers result counts and unexpected messages during the run
to be dumped in a summary at the end. This is a pattern we'd like to repeat in
several other formatters as well. Rather than re-implementing, this creates a
handler class that does nothing but store the data. Formatters can then choose
how to format this data and when to print it.
MozReview-Commit-ID: HKtVr5PxfOy
--HG--
extra : rebase_source : 22789db1b2fea1e44f47ef1aa9b22b21a6e8649c
It looks like the main cause of intermittent failures in getDirectory is
that the adb pull command fails because the emulator has hung. For other
commands, we usually handle this by checking the return code and raising
DMError if anything fails. There is mozharness/taskcluster code in
place to automatically retry tasks that throw DMError.
It looks like the main cause of intermittent failures in getDirectory is
that the adb pull command fails because the emulator has hung. For other
commands, we usually handle this by checking the return code and raising
DMError if anything fails. There is mozharness/taskcluster code in
place to automatically retry tasks that throw DMError.
Calling shutdown() causes two things to happen:
1) A 'shutdown' action is implicitly logged so handlers/formatters
can do things on log shutdown.
2) Further attempts to use the logger raises a LoggerShutdownError.
The shutdown() method is also implicitly called when exiting the context
manager.
MozReview-Commit-ID: LLNojVoCBZY
--HG--
extra : rebase_source : db483da27e82971ade4b8e424f14694fabd050f1
Calling shutdown() causes two things to happen:
1) A 'logger_shutdown' action is implicitly logged so handlers/formatters
can do things on log shutdown.
2) Further attempts to use the logger raise a LoggerShutdownError.
The shutdown() method is also implicitly called when the StructuredLogger's
destructor is run, or when exiting a context manager.
MozReview-Commit-ID: LLNojVoCBZY
--HG--
extra : rebase_source : 373b7e70f6a2121d29d7deccfe9bf4cc0f402e3b
This method has not a single caller and as such doesn't seem to
be necessary anymore.
MozReview-Commit-ID: qhNK3EBc6Q
--HG--
extra : rebase_source : 2978829739f0bc465f98b8f6b727c27a03a42b11
Reading the whole zip entry into memory is inefficient and can cause
OOMs if the entry is large enough. Let the ZipFile object choose the
most efficient extraction strategy instead.
The code in |mach test| for test resolving, should get merged with the TestResolver
class in moztest.resolve. This way it can be shared with other modules and we'll
have a single canonical place for all our test resolving logic.
MozReview-Commit-ID: IHRXXi5mB4G
--HG--
extra : rebase_source : 6f96d06412ab8fa152ac5d9bdd15acbcdc9695c4
The TestMetadata and TestResolver classes aren't technically part of the build
system. The only connection is that they consume some build system output.
The next patch in this series is going to be merging in a bunch of other test
resolving logic from other parts of the tree. Moving this out first allows us
to keep that extra logic out of mozbuild.
MozReview-Commit-ID: 1eq4SjFVCyW
--HG--
rename : python/mozbuild/mozbuild/test/test_testing.py => testing/mozbase/moztest/tests/test_resolve.py
extra : rebase_source : 7ff11f9ec455547533082d20cb5371845f7a4f21
Variable appDir was being referenced before assignment. Changed the try-except-finally blocks to handle the error.
MozReview-Commit-ID: AHEeVhmPfQI
--HG--
extra : rebase_source : b0dd78f3895bb34c4e916bc0441dd9ae5e643dfc
Minor note:
reftests should've turned off uploadEnabled in the first place.
reftests should have unified telemetry on. It's the future.
MozReview-Commit-ID: 9spzuUAXwwP
This fixes an exception when a test_status/test_end is logged before a
suite_start. This case should be an error anyway, but might as well fix this to
keep the logs looking clean.
MozReview-Commit-ID: 2TlWlSmczwH
--HG--
extra : rebase_source : c33aed0870d7b7fa51d855383d6336331d4f22fc
This just adds two basic tests, one for a passing test and another for a
failing one. In mochitest, we use privileged APIs to also tests crashes,
assertions, asan and leaks. But these APIs aren't available to reftests
so I'm not sure how we can test these things.
I figure it's not worth holding the framework up on this though, I'll file
a follow-up to figure out something to do for that.
MozReview-Commit-ID: 59TSbsugT5T
--HG--
extra : rebase_source : 72ecd817017c8b7d55eab879db4f6ad5fecc54c0
This includes code for downloading a Firefox binary, downloading + setting up a tests.zip and
running output through mozharness' output parsers. This is all stuff that will also be required
for the reftest selftests.
I couldn't think of a better location to put this stuff, suggestions welcome.
MozReview-Commit-ID: 59TSbsugT5T
--HG--
extra : rebase_source : a328f6bc90e73fe23f9054933cd01a30065419f6
dm.killProcess correctly tries to use 'am force-stop' in preference to kill()
to end a process. But some clients of killProcess specify a kill signal and
use killProcess for purposes other than endding the process, for example, to trigger
crash dumps. To allow for these cases, it is better to not use force-stop when a
signal is specified.
This switches most tests over to use pytest as the runner instead of unittest (taking
advantage of the fact that pytest can run unittest based tests).
There were a couple tests that had failures when swithing to pytest:
config/tests/unit-expandlibs.py
xpcom/idl-parser/xpidl/runtests.py
For these tests, I added a runwith='unittest' argument so that they still run the
same way as before. Once we fix them to use pytest, the unittest logic in mozunit.py
can be deleted.
MozReview-Commit-ID: Gcsz6z8MeOi
--HG--
extra : rebase_source : 3c762422ce0af54cbbe7d9fc20085a2d1ebe7057
The poll() call in SystemResourceMonitor.stop might fail even though
there is something to read from the pipe, in some corner cases, and
python won't let us know about it. In that case, an exception is thrown,
leaving the SystemResourceMonitor (and its callers) in a weird state. In
practice, this leads BuildMonitor.__exit__ to recall stop, which then
fails.
So when poll() throws an exception, we pretend there's still something
to read, and we try to read anyways. If there is something to read,
recv() will return it, otherwise, it will throw an exception of its own,
which we catch, pretending we're done.
Furthermore, when there is nothing to read from the pipe, poll() simply
returns False, and our loop never sets `done` to True, and we then hit
an assert, which doesn't have its place here, so we remove it.
Finally, the other end of the pipe might have died at any time, making
sending over the pipe fail, so we also protect against that.
With all these changes, it feels like the reason to backout bug 1239939
in bug 1272782 should have been dealt with, and we can drop the timeout
again.
--HG--
extra : rebase_source : ac72dd5b2602cf3ffddfb429f95e02380f939893
The poll() call in SystemResourceMonitor.stop might fail even though
there is something to read from the pipe, in some corner cases, and
python won't let us know about it. In that case, an exception is thrown,
leaving the SystemResourceMonitor (and its callers) in a weird state. In
practice, this leads BuildMonitor.__exit__ to recall stop, which then
fails.
So when poll() throws an exception, we pretend there's still something
to read, and we try to read anyways. If there is something to read,
recv() will return it, otherwise, it will throw an exception of its own,
which we catch, pretending we're done.
Furthermore, when there is nothing to read from the pipe, poll() simply
returns False, and our loop never sets `done` to True, and we then hit
an assert, which doesn't have its place here, so we remove it.
Finally, the other end of the pipe might have died at any time, making
sending over the pipe fail, so we also protect against that.
With all these changes, it feels like the reason to backout bug 1239939
in bug 1272782 should have been dealt with, and we can drop the timeout
again.
--HG--
extra : rebase_source : fededf989fe9021654b67d5a070f7e49aa717f3c
Currently manifestparser will only look for line continuations *after* looking for a key. This means
that line continuations cannot contain key separators. For example, this:
[test]
foo=
bar=baz
gets treated as:
{'name': 'test', 'foo': '', 'bar': 'baz'}
Here, bar=baz will be treated as a new key/value pair despite the indentation. This patch switches
the order around, so we look for a continuation first. Now, it is only treated as a continuation if
the indent is greater than the indent of the preceding key.
So this manifest:
[test]
foo=bar
baz=fleem
is a continuation and results in:
{'name': 'test', 'foo': 'bar\nbaz=fleem'}
But this manifest:
[test]
foo=bar
baz=fleem
is not a continuation, and yields:
{'name': 'test', 'foo': 'bar', 'baz': 'fleem'}
MozReview-Commit-ID: FAMP5TUIo9q
--HG--
extra : rebase_source : 624c53cfe0565374c1224dd86a3fffc8831279d3
The -fsanitize=integer analysis from UBSan can be helpful to detect signed and unsigned integer overflows in the codebase. Unfortunately, those occur very frequently, making it impossible to test anything with it without the use of a huge blacklist. This patch includes a blacklist that is broad enough to silence everything that would drain performance too much. But even with this blacklist, neither tests nor fuzzing is "clean". We can however in the future combine this with static analysis to limit ourselves to interesting places to look at, or improve the dynamic analysis to omit typical benign overflows.
It also adds another attribute that can be used on functions. It is not used right now because it was initially easier to add things to the compile-time blacklist to get started.
Finally, it includes a runtime suppression list and patches various parts in the test harnesses to support that. It is currently empty and it should not be used on frequent overflows because it is expensive. However, it has the advantage that it can be used to differentiate between signed and unsigned overflows while the compile-time blacklist cannot do that. So it can be used to e.g. silence unsigned integer overflows on a file or function while still reporting signed issues. We can also use this suppression list for any other UBSan related suppressions, should we ever want to use other features from that sanitizer.
MozReview-Commit-ID: C5ofhfJdpCS
--HG--
extra : rebase_source : 952043a441b41b2f58ec4abc51ac15fa71fc142f
This is needed before we can upgrade to flake8 3.3.0, as that version starts flagging these errors.
These files were modified by running:
autopep8 --select E305 --in-place -r <dir>
on the affected directories. I did it one dir at a time and verified the result after each.
MozReview-Commit-ID: FmlsfiKIbtr
--HG--
extra : rebase_source : 9df32258cadff5d27a0e72113c57f782756c0b18
We've seen a couple of enormous log files (200MB+). This largely happened due to a bug in the test harness that resulted
in suite_start being called over and over again. In each of these instances, mozlog logged something like:
log.error("Suite start called multiple times: {}".format(data))
The problem is that 'data' contained every single test id in the suite, which was *a lot*. Dumping all test ids in that
error message is not useful for debugging. This patch limits the size of the 'suite_start' data in the error message to
100 characters.
MozReview-Commit-ID: GnPizNrZ2QJ
--HG--
extra : rebase_source : 985d484544da9ea4cce445ce406fe085f86f112b
This is case that got hit with the new mochitest selftest harness. In this scenario, several MochitestDesktop instances
(which call commandline.setup_logging in their constructor) are instantiated in the same interpreter. Because mozlog
implicitly saves the logger state, this meant that setup_logging kept appending duplicate handlers to the existing ones.
I believe that the intent of 'setup_logging' is to get a brand new logger, so it should ensure logger state is reset
on subsequent calls.
MozReview-Commit-ID: Jqyhbj7nC6z
--HG--
extra : rebase_source : f267489bef99f3ac3d657357002a0001610a038f
This is a minor convenience for downloading the installer from a url. It uses requests (which should be
available everywhere in-tree).
MozReview-Commit-ID: 8IfiVkYNr06
--HG--
extra : rebase_source : cb8798cf3adb61008a5dac3794043950e48c3d6a
This allows subprocesses to log to a shared stream via a queue, so that we
avoid the overhead of a multiprocessing Lock around all log access, but still
avoid races where two processes try to log simultaneously. It's mostly useful
where one process is responsible for the majority of logging, but some messages
will be generated in child processes.
MozReview-Commit-ID: ABl6cvpb6qI
--HG--
extra : rebase_source : 5c749074c1646c7abb865a71b31b3056137ef398
The devicemanager killProcess() is updated to use force-stop first, then
use kill if force-stop does not work.
Browser test harnesses are updated to check if killProcess() worked, and
warn if it failed.
Various minor improvements to aid debugging:
- recommend --verbose on most common failure
- in verbose mode, display platform
- in verbose mode, display file creation date of binaries
- in verbose mode, display sdk binary versions
- remind of x86 vs arm emulator and need for corresponding apk
sutagent is no longer built or used; devicemanagerSUT is completely
unused. After this change, devicemanagerADB is the only implementation of
devicemanager, and test harness options like --dm_trans are eliminated.
sutagent is no longer built or usedr; devicemanagerSUT is completely
unused. After this change, devicemanagerADB is the only implementation of
devicemanager, and the --dmTrans and similar options have been removed
from test harnesses and mach commands.
This fixes a regression introduced in Bug 1335873, which changes the mozbase
packages.txt to call mozlog's setup.py. Calling setup.py registers the
pytest_mozlog plugin for marionette-harness tests.
Instead, we can register the pytest-mozlog plugin via command-line arguments
to pytest, which are set in pytest.ini for the marionette-harness tests.
As a result, we can revert the mozbase packages.txt to not refer to mozlog's
setup.py
I'm leaving the pytest entry-point in mozlog's setup.py so that external
consumers don't have to register the pytest_mozlog plugin manually.
MozReview-Commit-ID: I5wNq5H1x3X
--HG--
extra : rebase_source : 614a47995bc1655f36053d2a05b08f94bfdbe476
To allow for pushing directories containing symbolic links, pushDir
now always copies the source directory to a temporary local copy
before pushing.
In addition, I have added error checking, so that pushDir will now fail
if its adb command fails and returns a non-0 status from _checkCmd.
I created a new Android 4.3 AVD and uploaded it to tooltool. This new
AVD is compatible with the "new" emulator included in recent versions
of the Android SDK (circa Android SDK Tools 25). To avoid destabilizing
the emulator automated tests run via taskcluster and mozharness, I'm
creating a new tooltool manifest for the new AVD and using it only from
mach android-emulator.
For consistency, I'm creating separate but identical manifests for x86,
renaming the mach-only 6.0 manifest, and deleting the old 2.3 manifest.
--HG--
rename : testing/config/tooltool-manifests/androidarm_6_0/releng.manifest => testing/config/tooltool-manifests/androidarm_6_0/mach-emulator.manifest
This simplifies the 'process_output' formatting in both the mach and tbpl formatters. It will also
add the string 'pid' somewhere in the format, but only if the process id is actually a positive int.
MozReview-Commit-ID: 6nc5q06cDfM
--HG--
extra : rebase_source : 2c9ef125c19c06942a0c39d783151e8aa486a92c
This formats the marionette-harness python tests to be a regular |mach python-test| suite. Though
we add subsuite=marionette, this is just for automation purposes. The new preferred way to run the
marionette harness tests locally is:
./mach python-test testing/marionette
They will also run if running the full suite.
The mozbase packages.txt file modifies mozlog to use 'setup.py' instead of 'pth'. The reason for
this is that the marionette-harness tests use the pytest_mozlog pytest plugin for formatting
their results (converts pytest format into something resembling the standard tbpl logging format).
In order for this plugin to get picked up however, mozlog's setup.py file needs to be processed.
MozReview-Commit-ID: Ata99evHxbd
--HG--
extra : rebase_source : 22382e3d65ce8454a1682cfced0d03477762e8fe
This gets raised when trying to run the marionette-harness python tests without an objdir.
It's safe to ignore because mozinfo.json will still be found via the 'dirs' variable which
gets passed in from the marionette harness.
MozReview-Commit-ID: Ata99evHxbd
--HG--
extra : rebase_source : 6e3910667146b9caf0a9abe9a707b34627ba272b
This simplifies the 'process_output' formatting in both the mach and tbpl formatters. It will also
add the string 'pid' somewhere in the format, but only if the process id is actually a positive int.
MozReview-Commit-ID: 6nc5q06cDfM
--HG--
extra : rebase_source : d2c84e838b8b76b1b607a3beace02843085fca26
This simplifies the 'process_output' formatting in both the mach and tbpl formatters. It will also
add the string 'pid' somewhere in the format, but only if the process id is actually a positive int.
MozReview-Commit-ID: 6nc5q06cDfM
--HG--
extra : rebase_source : 67a435b653eed3cd374037f4bd30e1f65bf5615a
This formats the marionette-harness python tests to be a regular |mach python-test| suite. Though
we add subsuite=marionette, this is just for automation purposes. The new preferred way to run the
marionette harness tests locally is:
./mach python-test testing/marionette
They will also run if running the full suite.
The mozbase packages.txt file modifies mozlog to use 'setup.py' instead of 'pth'. The reason for
this is that the marionette-harness tests use the pytest_mozlog pytest plugin for formatting
their results (converts pytest format into something resembling the standard tbpl logging format).
In order for this plugin to get picked up however, mozlog's setup.py file needs to be processed.
MozReview-Commit-ID: Ata99evHxbd
--HG--
extra : rebase_source : 16ed70edd38a53c3279d8632d7cba3df4d5216c3
This gets raised when trying to run the marionette-harness python tests without an objdir.
It's safe to ignore because mozinfo.json will still be found via the 'dirs' variable which
gets passed in from the marionette harness.
MozReview-Commit-ID: Ata99evHxbd
--HG--
extra : rebase_source : 10731561f644aa563c85ed4a8f70759e64eb4ed2
This modifies the errorsummary formatter to make use of the new keyed-by-group tests field in the suite_start
message.
MozReview-Commit-ID: 1lcw62fmofa
--HG--
extra : rebase_source : fb6e5b02ae7f6f9201625f4392c8f4f7629023c2
In suite_start, tests is currently a list of test ids. But to support a new 'test-centric' treeherder view, we need to
be able to map tests to their manifests somewhere in the structured log, and the suite_start tests field seemed like a
good spot.
Because not all test suites use manifests, this is being called a "group" so it could potentially be re-used with
directories, tags or subsuites.
MozReview-Commit-ID: 2Sc7nqJrWts
--HG--
extra : rebase_source : d999df4c137b4752f118017a5db337e2ba6b467d
Currently the List and Tuple DataTypes must specify what items they contain. But there's no way to specify
item types recursively, e.g List(Tuple(Int, Int)). Also the Dict type can't specify the item types it must
contain either. Dict is a special case because we may want to control both keys and values.
This patch formalizes a ContainerType (of which List, Tuple and Dict are subclasses).
MozReview-Commit-ID: Bouhy1DIAyD
--HG--
extra : rebase_source : e7b26f4411861fc3065b4b5b746f05172f70d455
Not all report objects are strings, tuples or have a .crashrepr
property. In particular some brokeness in wdspec test setup produced
an object with a .errorstring property. Handle that and the general
case of being passed a weird object more gracefully.
MozReview-Commit-ID: 8vfuNNmwjhC
--HG--
extra : rebase_source : d7dd7ed6f76353df3bc82ed5b5bbd7d29cc4828e
When using pytest-xdist we need to gather the tests for each slave. Lack of a better hook means I'm then using the first test start to log the suite_start message including the collected tests.
MozReview-Commit-ID: 7l22z9RvIhx
--HG--
extra : rebase_source : c5002956ae81585b47dc19c571a57a7641d8ce9b
Stripping the path from the test file could cause duplicates if two files had the same name in different directories. Splitting on '::' also causes an issue when test classes are used and there are too many values to unpack.
MozReview-Commit-ID: Ex5nHl3SGaQ
--HG--
extra : rebase_source : 13198d8a886928402b6b079c441e8b1d675ebfa1
If the optional pytest-metadata plugin is installed, add the metadata as run_info when logging the suite_start to give additional context to the results.
MozReview-Commit-ID: 1MeXZTmzvyp
--HG--
extra : rebase_source : e5f9fd4fddc2bbe2cc8fbf76b4f2a5034a14e722
Unexpected passes can cause failures in pytest when the expected failures are set to strict. When we're logging these we want to indicate that the correct outcome.
MozReview-Commit-ID: 8VoR6CTrynY
--HG--
extra : rebase_source : 0624129eab8034bb2aa6b389216f701b28bd89f9
Include failure messages and stack traces even when failures are expected.
MozReview-Commit-ID: HSNYfbTEH9O
--HG--
extra : rebase_source : e9117fb75d056284a7427796db9a6ba47ce1f53f
When using pytest-xdist to run tests in parallel, failures are serialised as strings. This means we're unable to reliably extract the message and line number, so instead we log the stack and the message as the serialised string result.
MozReview-Commit-ID: 6vrEjBtkXK8
--HG--
extra : rebase_source : f337653f90cbbe6b1972b7d1e5e1a222c0a4604f
The taskcluster docker image for source-check tasks does not have 'ifconfig' installed. We
could add this package, but ifconfig is more or less deprecated in favour of 'ip addr show'.
Although the formats of both commands are different, because the test pulls ip addresses out
of the output with regexes, the only change that is needed to make sure the tests still pass
is to change the command.
MozReview-Commit-ID: 758Qb6KSHzS
--HG--
extra : rebase_source : 7ff59a448bfac6414a88adfac361463157ec5275
The subsuite is added conditionally because we only have the capability of
running source-check tasks on linux at the moment. Once taskcluster support
for windows and mac has matured a bit and the taskcluster configs support
source-check there, we should apply the subuite unconditionally.
MozReview-Commit-ID: Kk9Irz3fn14
--HG--
extra : rebase_source : b9266a06583083c36477d4e93f5462ee614cdb71
This refactors some of the error logic into a new IniParseError exception. In addition to checking for a 'name' property, we also
check for a 'path' property. This is because some file-like objects coming out of the build system use a 'path' attribute instead
of the standard 'name'.
MozReview-Commit-ID: EXXl9gt1MQk
--HG--
extra : rebase_source : a130824e6724fdef5cf0bd627b19db68ec5fe683
Previously manifestparser only supported inline comments on conditional keys, such as
skip-if, fails-if, subsuite-if, etc. But on any other key name, inline comments weren't
supported. This was a bad situation because people saw these skip-if comments, and
naturally assumed inline comments worked everywhere. Then they were surprised when things
broke in mysterious ways.
This patch removes the special-casing for skip-if and friends and allows inline comments
everywhere. A caveat to this, is that ' #' is no longer a valid substring in a value.
MozReview-Commit-ID: Hr0BIwzTgaJ
--HG--
extra : rebase_source : f9a042da2137f200434140a7e472df0799f6606d
It turns out there are shockingly few cases of manifestparser manifests that actually use the ';'
character as a comment. Because we will soon allow inline comments, deprecating the use of ';' will
ensure that values are allowed to have semicolons in them.
Even without inline comments, might as well enforce consistency across manifests.
MozReview-Commit-ID: AEPPQFdNXG0
--HG--
extra : rebase_source : 3540fa385f328bffb020c0a6debc4d2b3a90ed39
On architectures like alpha and ia64, the glibc does not use the
canonical ABI version number 6 but 6.1 and therefore the filename
of the C library is not libc.so.6 but libc.so.6.1. We're therefore
making the Python code more flexible and use find_library from
ctypes.util to determine the filename from the environment instead
of hard-coding it.
--HG--
extra : rebase_source : 64676648cec9975045a6dfeae1cfc9213226e242
ensuring moz.build files have BUG_COMPONENTS for all files
in the testing/ subdirectory is a win. There are a lot of
older files and some files used in many harnesses. If the
files are primarily used for mochitest, they are associated
with the testing::mochitest component, otherwise I chose
the testing::general component.
There is an exception web-platform-tests, these have many
test files that need to be matched to proper components.
MozReview-Commit-ID: IIv9W2kEqeN
Registered callback handlers for tests should receive the correct
test status when the test has been finished, and not always "Error".
This change allows those callbacks to run specific code for individual
test results, eg. only do screenshots for failures.
MozReview-Commit-ID: FfbCRR0Jvjb
--HG--
extra : rebase_source : c73253acbb666ca62b23f806145c20f0a70c934c
Registered callback handlers for tests should receive the correct
test status when the test has been finished, and not always "Error".
This change allows those callbacks to run specific code for individual
test results, eg. only do screenshots for failures.
MozReview-Commit-ID: FfbCRR0Jvjb
--HG--
extra : rebase_source : 98c69eea450f35312fd43bb7237a9d00e90636c4
This fixes a regression when switching to the StructuredOutputParser in mozharness.
Previously, mozleak was relying on the string "TEST-UNEXPECTED-FAIL" to turn the jobs
orange, but it was doing so at the "warning" level. The StructuredOutputParser requires
the "error" level to set the appropriate tbpl_status.
MozReview-Commit-ID: 9u9mwqrkA6E
--HG--
extra : rebase_source : d31ece08232a87713702e713076071fa10fb8324
This fixes a regression when switching to the StructuredOutputParser in mozharness.
Previously, mozleak was relying on the string "TEST-UNEXPECTED-FAIL" to turn the jobs
orange, but it was doing so at the "warning" level. The StructuredOutputParser requires
the "error" level to set the appropriate tbpl_status.
MozReview-Commit-ID: 9u9mwqrkA6E
--HG--
extra : rebase_source : 4d975e481c8257b178a145497bc53eabc28ed182
This is intended as a structured replacement for the assertion checks
that previously used unstructured logs. It adds a log action
assertion_count, which takes the actual number of assertions observed
during a test, the minimum expeced number and the maximum expected
number. It also updates the reftest harness to use this logging.
MozReview-Commit-ID: JgjLlaYuvSG
The only consumer of `mozrunner.devices.base.Device.setup_port_forward`
was Marionette, which now uses `mozdevice.DeviceManagerADB.forward`
directly.
MozReview-Commit-ID: 72ROrOixKvM
--HG--
extra : rebase_source : f998e6c37161f851da450bd98ee27ba04a50f16f
This deprecates PYTHON_UNIT_TESTS and replaces it with PYTHON_UNITTEST_MANIFESTS.
In the build system, this means python unittests will be treated the same as all
other test suites that use manifestparser. New manifests called 'python.ini' have
been created for all test directories containing python unittests.
MozReview-Commit-ID: IBHG7Thif2D
--HG--
extra : rebase_source : 11a92a2bc544d067946bbd774975140147458caa
The mozbase unittests don't use mozunit, so their output is confusing in the log.
This makes mozbase output consistent with the rest of the python unittests.
MozReview-Commit-ID: AIs5mza8Rn6
--HG--
extra : rebase_source : 10f65e612f5b3cebb921c47699f5a8be7cd2ba5a
The mozsystemmonitor test is under mozsystemmonitor/mozsystemmonitor/test instead of
mozsystemmonitor/tests like all the other mozbase modules.
MozReview-Commit-ID: AIs5mza8Rn6
--HG--
rename : testing/mozbase/mozsystemmonitor/mozsystemmonitor/test/test_resource_monitor.py => testing/mozbase/mozsystemmonitor/tests/test_resource_monitor.py
extra : rebase_source : dde714fb9212f19d1f8ba566f574bd7e9d7c4030
For some reason calling os.getpgid(proc.pid) in this bug results in an OSError "No such process"
on OSX. This bug is starting the ProcessHandler from a concurrent.futures Thread, that must be
somehow related.
I tried debugging this, but couldn't figure out why this is happening. However, the pgid is not
needed for this use case, and simply ignoring the error works. We also ignore this very same
exception when calling os.getpgid elsewhere in mozprocess, so there must be some weird OSXism
happening.
MozReview-Commit-ID: 2YXhBaORC5s
--HG--
extra : rebase_source : 120e4bff7ef29d2a0ad1e3bdd2df11b8b682d981
Adapt check_for_crashes() for latest mozcrash changes, which returns the number of crashes for both log_crashes()
and check_for_crashes() now. Also the runner should accumulate the number of crashes so it will be known at any
time how many times the process has been crashed.
MozReview-Commit-ID: Dl9FlH1TalH
--HG--
extra : rebase_source : b27895482fcad099cf4fcfc01a65fe0fbc5243e3
Currently check_for_crashes() behaves differently compared to log_crashes(), whereby it only returns a
boolean if a crash has been detected but not the amount of crash reports found. We should make sure that
both methods behave the same. Given that this change might affect consumers, we should have a major version
bump for the new release.
MozReview-Commit-ID: LiPaozJL5NF
--HG--
extra : rebase_source : d4392207399a1383a20e037bcf73f44bf3c36c7d
This causes consumers managing defaults themselves to fail to find a default
subsuite for tests, because the manifestparser will have provided a blank
default value by the time they incorporate defaults into a test definition.
This patch removes the provided defaults and updates a number of places assuming
the 'subsuite' field is always present.
MozReview-Commit-ID: 1jPy52VmEPr
This causes consumers managing defaults themselves to fail to find a default
subsuite for tests, because the manifestparser will have provided a blank
default value by the time they incorporate defaults into a test definition.
This patch removes the provided defaults and updates a number of places assuming
the 'subsuite' field is always present.
MozReview-Commit-ID: 1jPy52VmEPr
--HG--
extra : rebase_source : be7a2504af6853abb1bc532a058738f33d8dcbee
added testing/mozbase to tools/lint/flake8.lint
fixed a first batch of PEP8 errors/warnings
at first the commad autopep8 -i --max-line-length 99 -r -j 8 .
has been used to fix simpler problems, run from testing/mozbase
some of the issues can not easily fixed :
- undefined 'names' in code for example isLinux - isLinux and isBsd "fixed" with # noqa
- undefined 'message' resolved with return fmt.format(...
- undefined 'structured' resolved replacing those with mozlog
- long comments - some remaining - addressed with # noqa
- package level import everything - addressed with # flake8: noqa
restored testing/mozbase/mozdevice/mozdevice/Zeroconf.py
fixed issues reported on mozreview
fixed ')' in testing/mozbase/mozprocess/mozprocess/qijo.py imports
finally fixed multiline string at testing/mozbase/manifestparser/tests/test_manifestparser.py:114
^^^ and again, but now with ./mach python-test --path-only testing/mozbase/manifestparser/tests/test_manifestparser.py passing
fixed testing/mozbase/manifestparser/tests/test_convert_directory.py assert
fixed this error:
10:15:21 INFO - return lambda line: stack_fixer_module.fixSymbols(line)
10:15:21 INFO - TypeError: fixSymbols() takes exactly 2 arguments (1 given)
fixed two spaces lint error even of # noqa comments
restored assignement to lambda with # noqa to silence the lint error
global noqa for testing/mozbase/manifestparser/tests/test_filters.py
stupid is/is not error...
MozReview-Commit-ID: 1FpJF54GqIi
--HG--
extra : rebase_source : 3cf0277fb36a296e3506aeacc2ff05e1b03f9eac
Andreas Tolfsen
Dustin J. Mitchell
Andrew Halberstadt
Bob Clary
Geoff Brown
Sylvestre Ledru
Sebastian Hengst
Henrik Skupin
Vedant Chakravadhanula
Cosmin Sabou
James Graham
William Lachance
Dorel Luca
Mike Hommey
Csoregi Natalia
Brindusan Cristian
Tom Prince
Dave Hunt
Dan Banner
Tarek Ziadé
Ryan VanderMeulen
Nathan Froyd
Andrew Swan
Chris H-C
Phil Ringnalda
Attila Craciun
Steve Armand
Kartikaya Gupta
Myk Melez
Cosm
Ganti Sai Sarath Chandra
Wes Kocher
Eric Rahm
Alex Gaynor
Christian Holler
Carsten "Tomcat" Book
Dão Gottwald
Ting-Yu Chou
Maja Frydrychowicz
Andrea Marchesini
Iris Hsiao
James Graham
Sam Giles
Jim Chen
Florian Quèze
Brad Lassey
John Paul Adrian Glaubitz
Joel Maher
Andreas Tolfsen
Julian Seward
Kris Maglione
Ehsan Akhgari
Chris Manchester
Randell Jesup
Gregory Szorc
Ted Mielczarek
David Anderson
Francesco Pischedda