This is intended as a structured replacement for the assertion checks
that previously used unstructured logs. It adds a log action
assertion_count, which takes the actual number of assertions observed
during a test, the minimum expeced number and the maximum expected
number. It also updates the reftest harness to use this logging.
MozReview-Commit-ID: JgjLlaYuvSG
The mozbase unittests don't use mozunit, so their output is confusing in the log.
This makes mozbase output consistent with the rest of the python unittests.
MozReview-Commit-ID: AIs5mza8Rn6
--HG--
extra : rebase_source : 10f65e612f5b3cebb921c47699f5a8be7cd2ba5a
added testing/mozbase to tools/lint/flake8.lint
fixed a first batch of PEP8 errors/warnings
at first the commad autopep8 -i --max-line-length 99 -r -j 8 .
has been used to fix simpler problems, run from testing/mozbase
some of the issues can not easily fixed :
- undefined 'names' in code for example isLinux - isLinux and isBsd "fixed" with # noqa
- undefined 'message' resolved with return fmt.format(...
- undefined 'structured' resolved replacing those with mozlog
- long comments - some remaining - addressed with # noqa
- package level import everything - addressed with # flake8: noqa
restored testing/mozbase/mozdevice/mozdevice/Zeroconf.py
fixed issues reported on mozreview
fixed ')' in testing/mozbase/mozprocess/mozprocess/qijo.py imports
finally fixed multiline string at testing/mozbase/manifestparser/tests/test_manifestparser.py:114
^^^ and again, but now with ./mach python-test --path-only testing/mozbase/manifestparser/tests/test_manifestparser.py passing
fixed testing/mozbase/manifestparser/tests/test_convert_directory.py assert
fixed this error:
10:15:21 INFO - return lambda line: stack_fixer_module.fixSymbols(line)
10:15:21 INFO - TypeError: fixSymbols() takes exactly 2 arguments (1 given)
fixed two spaces lint error even of # noqa comments
restored assignement to lambda with # noqa to silence the lint error
global noqa for testing/mozbase/manifestparser/tests/test_filters.py
stupid is/is not error...
MozReview-Commit-ID: 1FpJF54GqIi
--HG--
extra : rebase_source : 3cf0277fb36a296e3506aeacc2ff05e1b03f9eac
This allows the logs to work with the structured reftest viewer.
MozReview-Commit-ID: CY71vSdDjLP
--HG--
extra : rebase_source : 6b83d98aff1c5e73ac0a802b5a83b8be95adf59a
Add the `pytest_mozlog` module (from https://github.com/davehunt/pytest-mozlog)
to mozbase/mozlog/mozlog, so that we can use it in the logging of the
Marionette harness tests.
Add an entry point for the plugin to mozlog/setup.py so that
when mozlog is installed, pytest will register the pytest_mozlog
plugin.py module as the plugin 'mozlog'.
Make the following modifications to pytest_mozlog/plugin.py:
- Disable stdout logging by default in pytest-mozlog
In the pytest-mozlog plugin, disable logging to stdout by
default when setting up the mozlog logger, so that the
default output seen when using the plugin is just the
Pytest terminal output. The user can specify a mozlog
logger to write to stdout by using the '--log-* -' option,
in which case both Pytest and pytest-mozlog will log to stdout.
- Record error msg and lineno in pytest-mozlog logs
Extract the error message and line number for tests with failures/
errors from the Pytest test report, and pass these to the Mozlog
logs output by the pytest-mozlog plugin. For skipped tests,
pass the skip reason, and for unexpected passes allow Mozlog to
report its default message "expected FAIL".
- Report collected item count in pytest-mozlog
Add/modify Pytest hooks in the pytest-mozlog plugin so that the
suite_start event sent to the logger contains the correct list
of tests to be run, i.e. items collected by Pytest, instead
of an empty list. This allows mozlog to log the correct number
of tests to be run.
- Log skipped/xfailed tests with corresponding status (not 'PASS')
Log skipped tests with status 'SKIP' and expected failures
that do indeed fail with status 'FAIL'/expected 'FAIL'
so that these can be distinguished from true passing tests.
- Pass parsed arguments directly to mozlog.commandline.setup_logging
Pass the entire Pytest config's argument namespace to `setup_logging`
instead of copying over only the non-None formatters/formatter option
arguments. Use the new `allow_unused_options` parameter to stop
Mozlog from raising an error about options for unused formatters.
MozReview-Commit-ID: 4WpprytS5cI
--HG--
extra : rebase_source : 409de7a89e860471870e3adfd7634bc06f6e1873
Add a parameter `allow_unused_options` to the `setup_logging` and
`setup_handlers` methods in mozlog.commandline. When this parameter
is True, do not raise a ValueError in `setup_handlers` if the
command line arguments include formatter options for unused formatters.
(Default value is False.)
This allows parsed arguments from e.g. the pytest-mozlog plugin
to be passed directly to `setup_logging`, without filtering out
unused formatter options.
MozReview-Commit-ID: 8el0oaT4y1X
--HG--
extra : rebase_source : 0e969a783b1754885e6c25d35c552134773f1b5e
Ensure that the first line of the log for failing tests is identical to
one that would be produced by the tbplformatter from mozlog. This means
that treeherder will be able to sucessfully cross-match error lines.
Also make the data structures used for storing screenshots identical
between reftests and web-platform-tests so that we can start to move
toward removing the reftest-specific codepath here.
MozReview-Commit-ID: FZQXLjj9Ejv
--HG--
extra : rebase_source : 3e9bf70ff7915817b8c02032c8da0d0a659e4661
Any exception which gets thrown by a log handler while test results are getting generated, should
not cause test harnesses to stop immediately. To achive that the exception details are written to
stderr and not propagated up the stack.
MozReview-Commit-ID: ChyYxApYSGx
--HG--
extra : rebase_source : 9fc3fe597061bedb1df2f5b8de1daa4bd127ea1e
There are two identical pairs of screenshots (test/reference pair) which are
png/base64 raw files generated from mozlog's HTML formatter. One pair is stored
in the img element to present the visual result; the other pair is stored as a
hyperlink source in the visual result's title.
After part1 patch, we may have one more pair. It appears that the hyperlinks of
the visual result's titles could be eliminated since they are visually closed to
the visual results, and clicking the visual results provides the exact same
function.
DONTBUILD (NPOTB)
MozReview-Commit-ID: 4CLfYXX8g69
--HG--
extra : rebase_source : d4d3b4e08b66e737d75a2ca21b6e84b344a29fc8
Current mozlog (v3.2) doesn't support screenshot logs exported from wptrunner.
Add this support so we could run css test with --log-html to see more detail
information, such as screenshots of test/reference pages.
DONTBUILD (NPOTB)
MozReview-Commit-ID: AUJwYfvNfda
--HG--
extra : rebase_source : 1d3d1fcdc396638d256336d3dfaf5ba5bad35168
Structured logs bring many benefits. We can stop parsing the logs for magic strings, we
can modify the format without breaking things, and we can stream results into systems like
ActiveData. The structured logs originate primarily in reftest.js. StructuredLog.jsm is
used to generate the JSON-based log stream. Finally OutputHandler in the python harness
reads structured output from stdout, and formats it into human readable form.
MozReview-Commit-ID: G3ZLkMRl6p7
--HG--
extra : commitid : J3ui9XYWR3Q
extra : rebase_source : 77ed0ba842cc8e557141fb3494212b06868c728a
extra : amend_source : 735d48225a2e627e0fe45fc11b50b6c49a885a4b
extra : source : d1779fe421c3c7cd8e3d191816776390dc104f37
Structured logs bring many benefits. We can stop parsing the logs for magic strings, we
can modify the format without breaking things, and we can stream results into systems like
ActiveData. The structured logs originate primarily in reftest.js. StructuredLog.jsm is
used to generate the JSON-based log stream. Finally OutputHandler in the python harness
reads structured output from stdout, and formats it into human readable form.
--HG--
extra : commitid : J3ui9XYWR3Q
extra : rebase_source : 6bae978126dbd5beddc39332c7cbce0c1354cd87
extra : amend_source : 735d48225a2e627e0fe45fc11b50b6c49a885a4b
James Graham
Andrew Halberstadt
Francesco Pischedda
Anjana Vakil
Jeremy Chen
Henrik Skupin
Shing Lyu
Julian Seward
Phil Ringnalda
Julien Pagès
Wes Kocher
Manish Goregaokar
Nick Alexander
Hiroyuki Ikezoe
Dave Hunt
Maciek