This fixes an exception when a test_status/test_end is logged before a
suite_start. This case should be an error anyway, but might as well fix this to
keep the logs looking clean.
MozReview-Commit-ID: 2TlWlSmczwH
--HG--
extra : rebase_source : c33aed0870d7b7fa51d855383d6336331d4f22fc
This just adds two basic tests, one for a passing test and another for a
failing one. In mochitest, we use privileged APIs to also tests crashes,
assertions, asan and leaks. But these APIs aren't available to reftests
so I'm not sure how we can test these things.
I figure it's not worth holding the framework up on this though, I'll file
a follow-up to figure out something to do for that.
MozReview-Commit-ID: 59TSbsugT5T
--HG--
extra : rebase_source : 72ecd817017c8b7d55eab879db4f6ad5fecc54c0
This switches most tests over to use pytest as the runner instead of unittest (taking
advantage of the fact that pytest can run unittest based tests).
There were a couple tests that had failures when swithing to pytest:
config/tests/unit-expandlibs.py
xpcom/idl-parser/xpidl/runtests.py
For these tests, I added a runwith='unittest' argument so that they still run the
same way as before. Once we fix them to use pytest, the unittest logic in mozunit.py
can be deleted.
MozReview-Commit-ID: Gcsz6z8MeOi
--HG--
extra : rebase_source : 3c762422ce0af54cbbe7d9fc20085a2d1ebe7057
This is needed before we can upgrade to flake8 3.3.0, as that version starts flagging these errors.
These files were modified by running:
autopep8 --select E305 --in-place -r <dir>
on the affected directories. I did it one dir at a time and verified the result after each.
MozReview-Commit-ID: FmlsfiKIbtr
--HG--
extra : rebase_source : 9df32258cadff5d27a0e72113c57f782756c0b18
We've seen a couple of enormous log files (200MB+). This largely happened due to a bug in the test harness that resulted
in suite_start being called over and over again. In each of these instances, mozlog logged something like:
log.error("Suite start called multiple times: {}".format(data))
The problem is that 'data' contained every single test id in the suite, which was *a lot*. Dumping all test ids in that
error message is not useful for debugging. This patch limits the size of the 'suite_start' data in the error message to
100 characters.
MozReview-Commit-ID: GnPizNrZ2QJ
--HG--
extra : rebase_source : 985d484544da9ea4cce445ce406fe085f86f112b
This is case that got hit with the new mochitest selftest harness. In this scenario, several MochitestDesktop instances
(which call commandline.setup_logging in their constructor) are instantiated in the same interpreter. Because mozlog
implicitly saves the logger state, this meant that setup_logging kept appending duplicate handlers to the existing ones.
I believe that the intent of 'setup_logging' is to get a brand new logger, so it should ensure logger state is reset
on subsequent calls.
MozReview-Commit-ID: Jqyhbj7nC6z
--HG--
extra : rebase_source : f267489bef99f3ac3d657357002a0001610a038f
This allows subprocesses to log to a shared stream via a queue, so that we
avoid the overhead of a multiprocessing Lock around all log access, but still
avoid races where two processes try to log simultaneously. It's mostly useful
where one process is responsible for the majority of logging, but some messages
will be generated in child processes.
MozReview-Commit-ID: ABl6cvpb6qI
--HG--
extra : rebase_source : 5c749074c1646c7abb865a71b31b3056137ef398
This simplifies the 'process_output' formatting in both the mach and tbpl formatters. It will also
add the string 'pid' somewhere in the format, but only if the process id is actually a positive int.
MozReview-Commit-ID: 6nc5q06cDfM
--HG--
extra : rebase_source : d2c84e838b8b76b1b607a3beace02843085fca26
This simplifies the 'process_output' formatting in both the mach and tbpl formatters. It will also
add the string 'pid' somewhere in the format, but only if the process id is actually a positive int.
MozReview-Commit-ID: 6nc5q06cDfM
--HG--
extra : rebase_source : 67a435b653eed3cd374037f4bd30e1f65bf5615a
This simplifies the 'process_output' formatting in both the mach and tbpl formatters. It will also
add the string 'pid' somewhere in the format, but only if the process id is actually a positive int.
MozReview-Commit-ID: 6nc5q06cDfM
--HG--
extra : rebase_source : 2c9ef125c19c06942a0c39d783151e8aa486a92c
This modifies the errorsummary formatter to make use of the new keyed-by-group tests field in the suite_start
message.
MozReview-Commit-ID: 1lcw62fmofa
--HG--
extra : rebase_source : fb6e5b02ae7f6f9201625f4392c8f4f7629023c2
In suite_start, tests is currently a list of test ids. But to support a new 'test-centric' treeherder view, we need to
be able to map tests to their manifests somewhere in the structured log, and the suite_start tests field seemed like a
good spot.
Because not all test suites use manifests, this is being called a "group" so it could potentially be re-used with
directories, tags or subsuites.
MozReview-Commit-ID: 2Sc7nqJrWts
--HG--
extra : rebase_source : d999df4c137b4752f118017a5db337e2ba6b467d
Currently the List and Tuple DataTypes must specify what items they contain. But there's no way to specify
item types recursively, e.g List(Tuple(Int, Int)). Also the Dict type can't specify the item types it must
contain either. Dict is a special case because we may want to control both keys and values.
This patch formalizes a ContainerType (of which List, Tuple and Dict are subclasses).
MozReview-Commit-ID: Bouhy1DIAyD
--HG--
extra : rebase_source : e7b26f4411861fc3065b4b5b746f05172f70d455
Not all report objects are strings, tuples or have a .crashrepr
property. In particular some brokeness in wdspec test setup produced
an object with a .errorstring property. Handle that and the general
case of being passed a weird object more gracefully.
MozReview-Commit-ID: 8vfuNNmwjhC
--HG--
extra : rebase_source : d7dd7ed6f76353df3bc82ed5b5bbd7d29cc4828e
If the optional pytest-metadata plugin is installed, add the metadata as run_info when logging the suite_start to give additional context to the results.
MozReview-Commit-ID: 1MeXZTmzvyp
--HG--
extra : rebase_source : e5f9fd4fddc2bbe2cc8fbf76b4f2a5034a14e722
Unexpected passes can cause failures in pytest when the expected failures are set to strict. When we're logging these we want to indicate that the correct outcome.
MozReview-Commit-ID: 8VoR6CTrynY
--HG--
extra : rebase_source : 0624129eab8034bb2aa6b389216f701b28bd89f9
Include failure messages and stack traces even when failures are expected.
MozReview-Commit-ID: HSNYfbTEH9O
--HG--
extra : rebase_source : e9117fb75d056284a7427796db9a6ba47ce1f53f
When using pytest-xdist to run tests in parallel, failures are serialised as strings. This means we're unable to reliably extract the message and line number, so instead we log the stack and the message as the serialised string result.
MozReview-Commit-ID: 6vrEjBtkXK8
--HG--
extra : rebase_source : f337653f90cbbe6b1972b7d1e5e1a222c0a4604f
The subsuite is added conditionally because we only have the capability of
running source-check tasks on linux at the moment. Once taskcluster support
for windows and mac has matured a bit and the taskcluster configs support
source-check there, we should apply the subuite unconditionally.
MozReview-Commit-ID: Kk9Irz3fn14
--HG--
extra : rebase_source : b9266a06583083c36477d4e93f5462ee614cdb71
When using pytest-xdist we need to gather the tests for each slave. Lack of a better hook means I'm then using the first test start to log the suite_start message including the collected tests.
MozReview-Commit-ID: 7l22z9RvIhx
--HG--
extra : rebase_source : c5002956ae81585b47dc19c571a57a7641d8ce9b
Stripping the path from the test file could cause duplicates if two files had the same name in different directories. Splitting on '::' also causes an issue when test classes are used and there are too many values to unpack.
MozReview-Commit-ID: Ex5nHl3SGaQ
--HG--
extra : rebase_source : 13198d8a886928402b6b079c441e8b1d675ebfa1
This is intended as a structured replacement for the assertion checks
that previously used unstructured logs. It adds a log action
assertion_count, which takes the actual number of assertions observed
during a test, the minimum expeced number and the maximum expected
number. It also updates the reftest harness to use this logging.
MozReview-Commit-ID: JgjLlaYuvSG
The mozbase unittests don't use mozunit, so their output is confusing in the log.
This makes mozbase output consistent with the rest of the python unittests.
MozReview-Commit-ID: AIs5mza8Rn6
--HG--
extra : rebase_source : 10f65e612f5b3cebb921c47699f5a8be7cd2ba5a
added testing/mozbase to tools/lint/flake8.lint
fixed a first batch of PEP8 errors/warnings
at first the commad autopep8 -i --max-line-length 99 -r -j 8 .
has been used to fix simpler problems, run from testing/mozbase
some of the issues can not easily fixed :
- undefined 'names' in code for example isLinux - isLinux and isBsd "fixed" with # noqa
- undefined 'message' resolved with return fmt.format(...
- undefined 'structured' resolved replacing those with mozlog
- long comments - some remaining - addressed with # noqa
- package level import everything - addressed with # flake8: noqa
restored testing/mozbase/mozdevice/mozdevice/Zeroconf.py
fixed issues reported on mozreview
fixed ')' in testing/mozbase/mozprocess/mozprocess/qijo.py imports
finally fixed multiline string at testing/mozbase/manifestparser/tests/test_manifestparser.py:114
^^^ and again, but now with ./mach python-test --path-only testing/mozbase/manifestparser/tests/test_manifestparser.py passing
fixed testing/mozbase/manifestparser/tests/test_convert_directory.py assert
fixed this error:
10:15:21 INFO - return lambda line: stack_fixer_module.fixSymbols(line)
10:15:21 INFO - TypeError: fixSymbols() takes exactly 2 arguments (1 given)
fixed two spaces lint error even of # noqa comments
restored assignement to lambda with # noqa to silence the lint error
global noqa for testing/mozbase/manifestparser/tests/test_filters.py
stupid is/is not error...
MozReview-Commit-ID: 1FpJF54GqIi
--HG--
extra : rebase_source : 3cf0277fb36a296e3506aeacc2ff05e1b03f9eac
This allows the logs to work with the structured reftest viewer.
MozReview-Commit-ID: CY71vSdDjLP
--HG--
extra : rebase_source : 6b83d98aff1c5e73ac0a802b5a83b8be95adf59a
Add the `pytest_mozlog` module (from https://github.com/davehunt/pytest-mozlog)
to mozbase/mozlog/mozlog, so that we can use it in the logging of the
Marionette harness tests.
Add an entry point for the plugin to mozlog/setup.py so that
when mozlog is installed, pytest will register the pytest_mozlog
plugin.py module as the plugin 'mozlog'.
Make the following modifications to pytest_mozlog/plugin.py:
- Disable stdout logging by default in pytest-mozlog
In the pytest-mozlog plugin, disable logging to stdout by
default when setting up the mozlog logger, so that the
default output seen when using the plugin is just the
Pytest terminal output. The user can specify a mozlog
logger to write to stdout by using the '--log-* -' option,
in which case both Pytest and pytest-mozlog will log to stdout.
- Record error msg and lineno in pytest-mozlog logs
Extract the error message and line number for tests with failures/
errors from the Pytest test report, and pass these to the Mozlog
logs output by the pytest-mozlog plugin. For skipped tests,
pass the skip reason, and for unexpected passes allow Mozlog to
report its default message "expected FAIL".
- Report collected item count in pytest-mozlog
Add/modify Pytest hooks in the pytest-mozlog plugin so that the
suite_start event sent to the logger contains the correct list
of tests to be run, i.e. items collected by Pytest, instead
of an empty list. This allows mozlog to log the correct number
of tests to be run.
- Log skipped/xfailed tests with corresponding status (not 'PASS')
Log skipped tests with status 'SKIP' and expected failures
that do indeed fail with status 'FAIL'/expected 'FAIL'
so that these can be distinguished from true passing tests.
- Pass parsed arguments directly to mozlog.commandline.setup_logging
Pass the entire Pytest config's argument namespace to `setup_logging`
instead of copying over only the non-None formatters/formatter option
arguments. Use the new `allow_unused_options` parameter to stop
Mozlog from raising an error about options for unused formatters.
MozReview-Commit-ID: 4WpprytS5cI
--HG--
extra : rebase_source : 409de7a89e860471870e3adfd7634bc06f6e1873
Add a parameter `allow_unused_options` to the `setup_logging` and
`setup_handlers` methods in mozlog.commandline. When this parameter
is True, do not raise a ValueError in `setup_handlers` if the
command line arguments include formatter options for unused formatters.
(Default value is False.)
This allows parsed arguments from e.g. the pytest-mozlog plugin
to be passed directly to `setup_logging`, without filtering out
unused formatter options.
MozReview-Commit-ID: 8el0oaT4y1X
--HG--
extra : rebase_source : 0e969a783b1754885e6c25d35c552134773f1b5e
Ensure that the first line of the log for failing tests is identical to
one that would be produced by the tbplformatter from mozlog. This means
that treeherder will be able to sucessfully cross-match error lines.
Also make the data structures used for storing screenshots identical
between reftests and web-platform-tests so that we can start to move
toward removing the reftest-specific codepath here.
MozReview-Commit-ID: FZQXLjj9Ejv
--HG--
extra : rebase_source : 3e9bf70ff7915817b8c02032c8da0d0a659e4661
Any exception which gets thrown by a log handler while test results are getting generated, should
not cause test harnesses to stop immediately. To achive that the exception details are written to
stderr and not propagated up the stack.
MozReview-Commit-ID: ChyYxApYSGx
--HG--
extra : rebase_source : 9fc3fe597061bedb1df2f5b8de1daa4bd127ea1e
There are two identical pairs of screenshots (test/reference pair) which are
png/base64 raw files generated from mozlog's HTML formatter. One pair is stored
in the img element to present the visual result; the other pair is stored as a
hyperlink source in the visual result's title.
After part1 patch, we may have one more pair. It appears that the hyperlinks of
the visual result's titles could be eliminated since they are visually closed to
the visual results, and clicking the visual results provides the exact same
function.
DONTBUILD (NPOTB)
MozReview-Commit-ID: 4CLfYXX8g69
--HG--
extra : rebase_source : d4d3b4e08b66e737d75a2ca21b6e84b344a29fc8
Current mozlog (v3.2) doesn't support screenshot logs exported from wptrunner.
Add this support so we could run css test with --log-html to see more detail
information, such as screenshots of test/reference pages.
DONTBUILD (NPOTB)
MozReview-Commit-ID: AUJwYfvNfda
--HG--
extra : rebase_source : 1d3d1fcdc396638d256336d3dfaf5ba5bad35168
Steve Armand
Andrew Halberstadt
Sebastian Hengst
Ganti Sai Sarath Chandra
Dave Hunt
James Graham
Carsten "Tomcat" Book
Geoff Brown
Iris Hsiao
James Graham
Florian Quèze
Francesco Pischedda
Anjana Vakil
Jeremy Chen
Henrik Skupin