The -fsanitize=integer analysis from UBSan can be helpful to detect signed and unsigned integer overflows in the codebase. Unfortunately, those occur very frequently, making it impossible to test anything with it without the use of a huge blacklist. This patch includes a blacklist that is broad enough to silence everything that would drain performance too much. But even with this blacklist, neither tests nor fuzzing is "clean". We can however in the future combine this with static analysis to limit ourselves to interesting places to look at, or improve the dynamic analysis to omit typical benign overflows.
It also adds another attribute that can be used on functions. It is not used right now because it was initially easier to add things to the compile-time blacklist to get started.
Finally, it includes a runtime suppression list and patches various parts in the test harnesses to support that. It is currently empty and it should not be used on frequent overflows because it is expensive. However, it has the advantage that it can be used to differentiate between signed and unsigned overflows while the compile-time blacklist cannot do that. So it can be used to e.g. silence unsigned integer overflows on a file or function while still reporting signed issues. We can also use this suppression list for any other UBSan related suppressions, should we ever want to use other features from that sanitizer.
MozReview-Commit-ID: C5ofhfJdpCS
--HG--
extra : rebase_source : 952043a441b41b2f58ec4abc51ac15fa71fc142f
This is needed before we can upgrade to flake8 3.3.0, as that version starts flagging these errors.
These files were modified by running:
autopep8 --select E305 --in-place -r <dir>
on the affected directories. I did it one dir at a time and verified the result after each.
MozReview-Commit-ID: FmlsfiKIbtr
--HG--
extra : rebase_source : 9df32258cadff5d27a0e72113c57f782756c0b18
We've seen a couple of enormous log files (200MB+). This largely happened due to a bug in the test harness that resulted
in suite_start being called over and over again. In each of these instances, mozlog logged something like:
log.error("Suite start called multiple times: {}".format(data))
The problem is that 'data' contained every single test id in the suite, which was *a lot*. Dumping all test ids in that
error message is not useful for debugging. This patch limits the size of the 'suite_start' data in the error message to
100 characters.
MozReview-Commit-ID: GnPizNrZ2QJ
--HG--
extra : rebase_source : 985d484544da9ea4cce445ce406fe085f86f112b
This is case that got hit with the new mochitest selftest harness. In this scenario, several MochitestDesktop instances
(which call commandline.setup_logging in their constructor) are instantiated in the same interpreter. Because mozlog
implicitly saves the logger state, this meant that setup_logging kept appending duplicate handlers to the existing ones.
I believe that the intent of 'setup_logging' is to get a brand new logger, so it should ensure logger state is reset
on subsequent calls.
MozReview-Commit-ID: Jqyhbj7nC6z
--HG--
extra : rebase_source : f267489bef99f3ac3d657357002a0001610a038f
This is a minor convenience for downloading the installer from a url. It uses requests (which should be
available everywhere in-tree).
MozReview-Commit-ID: 8IfiVkYNr06
--HG--
extra : rebase_source : cb8798cf3adb61008a5dac3794043950e48c3d6a
This allows subprocesses to log to a shared stream via a queue, so that we
avoid the overhead of a multiprocessing Lock around all log access, but still
avoid races where two processes try to log simultaneously. It's mostly useful
where one process is responsible for the majority of logging, but some messages
will be generated in child processes.
MozReview-Commit-ID: ABl6cvpb6qI
--HG--
extra : rebase_source : 5c749074c1646c7abb865a71b31b3056137ef398
The devicemanager killProcess() is updated to use force-stop first, then
use kill if force-stop does not work.
Browser test harnesses are updated to check if killProcess() worked, and
warn if it failed.
Various minor improvements to aid debugging:
- recommend --verbose on most common failure
- in verbose mode, display platform
- in verbose mode, display file creation date of binaries
- in verbose mode, display sdk binary versions
- remind of x86 vs arm emulator and need for corresponding apk
sutagent is no longer built or used; devicemanagerSUT is completely
unused. After this change, devicemanagerADB is the only implementation of
devicemanager, and test harness options like --dm_trans are eliminated.
sutagent is no longer built or usedr; devicemanagerSUT is completely
unused. After this change, devicemanagerADB is the only implementation of
devicemanager, and the --dmTrans and similar options have been removed
from test harnesses and mach commands.
This fixes a regression introduced in Bug 1335873, which changes the mozbase
packages.txt to call mozlog's setup.py. Calling setup.py registers the
pytest_mozlog plugin for marionette-harness tests.
Instead, we can register the pytest-mozlog plugin via command-line arguments
to pytest, which are set in pytest.ini for the marionette-harness tests.
As a result, we can revert the mozbase packages.txt to not refer to mozlog's
setup.py
I'm leaving the pytest entry-point in mozlog's setup.py so that external
consumers don't have to register the pytest_mozlog plugin manually.
MozReview-Commit-ID: I5wNq5H1x3X
--HG--
extra : rebase_source : 614a47995bc1655f36053d2a05b08f94bfdbe476
To allow for pushing directories containing symbolic links, pushDir
now always copies the source directory to a temporary local copy
before pushing.
In addition, I have added error checking, so that pushDir will now fail
if its adb command fails and returns a non-0 status from _checkCmd.
I created a new Android 4.3 AVD and uploaded it to tooltool. This new
AVD is compatible with the "new" emulator included in recent versions
of the Android SDK (circa Android SDK Tools 25). To avoid destabilizing
the emulator automated tests run via taskcluster and mozharness, I'm
creating a new tooltool manifest for the new AVD and using it only from
mach android-emulator.
For consistency, I'm creating separate but identical manifests for x86,
renaming the mach-only 6.0 manifest, and deleting the old 2.3 manifest.
--HG--
rename : testing/config/tooltool-manifests/androidarm_6_0/releng.manifest => testing/config/tooltool-manifests/androidarm_6_0/mach-emulator.manifest
This simplifies the 'process_output' formatting in both the mach and tbpl formatters. It will also
add the string 'pid' somewhere in the format, but only if the process id is actually a positive int.
MozReview-Commit-ID: 6nc5q06cDfM
--HG--
extra : rebase_source : 2c9ef125c19c06942a0c39d783151e8aa486a92c
This formats the marionette-harness python tests to be a regular |mach python-test| suite. Though
we add subsuite=marionette, this is just for automation purposes. The new preferred way to run the
marionette harness tests locally is:
./mach python-test testing/marionette
They will also run if running the full suite.
The mozbase packages.txt file modifies mozlog to use 'setup.py' instead of 'pth'. The reason for
this is that the marionette-harness tests use the pytest_mozlog pytest plugin for formatting
their results (converts pytest format into something resembling the standard tbpl logging format).
In order for this plugin to get picked up however, mozlog's setup.py file needs to be processed.
MozReview-Commit-ID: Ata99evHxbd
--HG--
extra : rebase_source : 22382e3d65ce8454a1682cfced0d03477762e8fe
This gets raised when trying to run the marionette-harness python tests without an objdir.
It's safe to ignore because mozinfo.json will still be found via the 'dirs' variable which
gets passed in from the marionette harness.
MozReview-Commit-ID: Ata99evHxbd
--HG--
extra : rebase_source : 6e3910667146b9caf0a9abe9a707b34627ba272b
This simplifies the 'process_output' formatting in both the mach and tbpl formatters. It will also
add the string 'pid' somewhere in the format, but only if the process id is actually a positive int.
MozReview-Commit-ID: 6nc5q06cDfM
--HG--
extra : rebase_source : d2c84e838b8b76b1b607a3beace02843085fca26
This simplifies the 'process_output' formatting in both the mach and tbpl formatters. It will also
add the string 'pid' somewhere in the format, but only if the process id is actually a positive int.
MozReview-Commit-ID: 6nc5q06cDfM
--HG--
extra : rebase_source : 67a435b653eed3cd374037f4bd30e1f65bf5615a
This formats the marionette-harness python tests to be a regular |mach python-test| suite. Though
we add subsuite=marionette, this is just for automation purposes. The new preferred way to run the
marionette harness tests locally is:
./mach python-test testing/marionette
They will also run if running the full suite.
The mozbase packages.txt file modifies mozlog to use 'setup.py' instead of 'pth'. The reason for
this is that the marionette-harness tests use the pytest_mozlog pytest plugin for formatting
their results (converts pytest format into something resembling the standard tbpl logging format).
In order for this plugin to get picked up however, mozlog's setup.py file needs to be processed.
MozReview-Commit-ID: Ata99evHxbd
--HG--
extra : rebase_source : 16ed70edd38a53c3279d8632d7cba3df4d5216c3
This gets raised when trying to run the marionette-harness python tests without an objdir.
It's safe to ignore because mozinfo.json will still be found via the 'dirs' variable which
gets passed in from the marionette harness.
MozReview-Commit-ID: Ata99evHxbd
--HG--
extra : rebase_source : 10731561f644aa563c85ed4a8f70759e64eb4ed2
This modifies the errorsummary formatter to make use of the new keyed-by-group tests field in the suite_start
message.
MozReview-Commit-ID: 1lcw62fmofa
--HG--
extra : rebase_source : fb6e5b02ae7f6f9201625f4392c8f4f7629023c2
In suite_start, tests is currently a list of test ids. But to support a new 'test-centric' treeherder view, we need to
be able to map tests to their manifests somewhere in the structured log, and the suite_start tests field seemed like a
good spot.
Because not all test suites use manifests, this is being called a "group" so it could potentially be re-used with
directories, tags or subsuites.
MozReview-Commit-ID: 2Sc7nqJrWts
--HG--
extra : rebase_source : d999df4c137b4752f118017a5db337e2ba6b467d
Currently the List and Tuple DataTypes must specify what items they contain. But there's no way to specify
item types recursively, e.g List(Tuple(Int, Int)). Also the Dict type can't specify the item types it must
contain either. Dict is a special case because we may want to control both keys and values.
This patch formalizes a ContainerType (of which List, Tuple and Dict are subclasses).
MozReview-Commit-ID: Bouhy1DIAyD
--HG--
extra : rebase_source : e7b26f4411861fc3065b4b5b746f05172f70d455
Not all report objects are strings, tuples or have a .crashrepr
property. In particular some brokeness in wdspec test setup produced
an object with a .errorstring property. Handle that and the general
case of being passed a weird object more gracefully.
MozReview-Commit-ID: 8vfuNNmwjhC
--HG--
extra : rebase_source : d7dd7ed6f76353df3bc82ed5b5bbd7d29cc4828e
When using pytest-xdist we need to gather the tests for each slave. Lack of a better hook means I'm then using the first test start to log the suite_start message including the collected tests.
MozReview-Commit-ID: 7l22z9RvIhx
--HG--
extra : rebase_source : c5002956ae81585b47dc19c571a57a7641d8ce9b
Stripping the path from the test file could cause duplicates if two files had the same name in different directories. Splitting on '::' also causes an issue when test classes are used and there are too many values to unpack.
MozReview-Commit-ID: Ex5nHl3SGaQ
--HG--
extra : rebase_source : 13198d8a886928402b6b079c441e8b1d675ebfa1
If the optional pytest-metadata plugin is installed, add the metadata as run_info when logging the suite_start to give additional context to the results.
MozReview-Commit-ID: 1MeXZTmzvyp
--HG--
extra : rebase_source : e5f9fd4fddc2bbe2cc8fbf76b4f2a5034a14e722
Unexpected passes can cause failures in pytest when the expected failures are set to strict. When we're logging these we want to indicate that the correct outcome.
MozReview-Commit-ID: 8VoR6CTrynY
--HG--
extra : rebase_source : 0624129eab8034bb2aa6b389216f701b28bd89f9
Include failure messages and stack traces even when failures are expected.
MozReview-Commit-ID: HSNYfbTEH9O
--HG--
extra : rebase_source : e9117fb75d056284a7427796db9a6ba47ce1f53f
When using pytest-xdist to run tests in parallel, failures are serialised as strings. This means we're unable to reliably extract the message and line number, so instead we log the stack and the message as the serialised string result.
MozReview-Commit-ID: 6vrEjBtkXK8
--HG--
extra : rebase_source : f337653f90cbbe6b1972b7d1e5e1a222c0a4604f
The taskcluster docker image for source-check tasks does not have 'ifconfig' installed. We
could add this package, but ifconfig is more or less deprecated in favour of 'ip addr show'.
Although the formats of both commands are different, because the test pulls ip addresses out
of the output with regexes, the only change that is needed to make sure the tests still pass
is to change the command.
MozReview-Commit-ID: 758Qb6KSHzS
--HG--
extra : rebase_source : 7ff59a448bfac6414a88adfac361463157ec5275
The subsuite is added conditionally because we only have the capability of
running source-check tasks on linux at the moment. Once taskcluster support
for windows and mac has matured a bit and the taskcluster configs support
source-check there, we should apply the subuite unconditionally.
MozReview-Commit-ID: Kk9Irz3fn14
--HG--
extra : rebase_source : b9266a06583083c36477d4e93f5462ee614cdb71
This refactors some of the error logic into a new IniParseError exception. In addition to checking for a 'name' property, we also
check for a 'path' property. This is because some file-like objects coming out of the build system use a 'path' attribute instead
of the standard 'name'.
MozReview-Commit-ID: EXXl9gt1MQk
--HG--
extra : rebase_source : a130824e6724fdef5cf0bd627b19db68ec5fe683
Previously manifestparser only supported inline comments on conditional keys, such as
skip-if, fails-if, subsuite-if, etc. But on any other key name, inline comments weren't
supported. This was a bad situation because people saw these skip-if comments, and
naturally assumed inline comments worked everywhere. Then they were surprised when things
broke in mysterious ways.
This patch removes the special-casing for skip-if and friends and allows inline comments
everywhere. A caveat to this, is that ' #' is no longer a valid substring in a value.
MozReview-Commit-ID: Hr0BIwzTgaJ
--HG--
extra : rebase_source : f9a042da2137f200434140a7e472df0799f6606d
It turns out there are shockingly few cases of manifestparser manifests that actually use the ';'
character as a comment. Because we will soon allow inline comments, deprecating the use of ';' will
ensure that values are allowed to have semicolons in them.
Even without inline comments, might as well enforce consistency across manifests.
MozReview-Commit-ID: AEPPQFdNXG0
--HG--
extra : rebase_source : 3540fa385f328bffb020c0a6debc4d2b3a90ed39
On architectures like alpha and ia64, the glibc does not use the
canonical ABI version number 6 but 6.1 and therefore the filename
of the C library is not libc.so.6 but libc.so.6.1. We're therefore
making the Python code more flexible and use find_library from
ctypes.util to determine the filename from the environment instead
of hard-coding it.
--HG--
extra : rebase_source : 64676648cec9975045a6dfeae1cfc9213226e242
ensuring moz.build files have BUG_COMPONENTS for all files
in the testing/ subdirectory is a win. There are a lot of
older files and some files used in many harnesses. If the
files are primarily used for mochitest, they are associated
with the testing::mochitest component, otherwise I chose
the testing::general component.
There is an exception web-platform-tests, these have many
test files that need to be matched to proper components.
MozReview-Commit-ID: IIv9W2kEqeN
Registered callback handlers for tests should receive the correct
test status when the test has been finished, and not always "Error".
This change allows those callbacks to run specific code for individual
test results, eg. only do screenshots for failures.
MozReview-Commit-ID: FfbCRR0Jvjb
--HG--
extra : rebase_source : c73253acbb666ca62b23f806145c20f0a70c934c
Registered callback handlers for tests should receive the correct
test status when the test has been finished, and not always "Error".
This change allows those callbacks to run specific code for individual
test results, eg. only do screenshots for failures.
MozReview-Commit-ID: FfbCRR0Jvjb
--HG--
extra : rebase_source : 98c69eea450f35312fd43bb7237a9d00e90636c4
This fixes a regression when switching to the StructuredOutputParser in mozharness.
Previously, mozleak was relying on the string "TEST-UNEXPECTED-FAIL" to turn the jobs
orange, but it was doing so at the "warning" level. The StructuredOutputParser requires
the "error" level to set the appropriate tbpl_status.
MozReview-Commit-ID: 9u9mwqrkA6E
--HG--
extra : rebase_source : d31ece08232a87713702e713076071fa10fb8324
This fixes a regression when switching to the StructuredOutputParser in mozharness.
Previously, mozleak was relying on the string "TEST-UNEXPECTED-FAIL" to turn the jobs
orange, but it was doing so at the "warning" level. The StructuredOutputParser requires
the "error" level to set the appropriate tbpl_status.
MozReview-Commit-ID: 9u9mwqrkA6E
--HG--
extra : rebase_source : 4d975e481c8257b178a145497bc53eabc28ed182
This is intended as a structured replacement for the assertion checks
that previously used unstructured logs. It adds a log action
assertion_count, which takes the actual number of assertions observed
during a test, the minimum expeced number and the maximum expected
number. It also updates the reftest harness to use this logging.
MozReview-Commit-ID: JgjLlaYuvSG
The only consumer of `mozrunner.devices.base.Device.setup_port_forward`
was Marionette, which now uses `mozdevice.DeviceManagerADB.forward`
directly.
MozReview-Commit-ID: 72ROrOixKvM
--HG--
extra : rebase_source : f998e6c37161f851da450bd98ee27ba04a50f16f
This deprecates PYTHON_UNIT_TESTS and replaces it with PYTHON_UNITTEST_MANIFESTS.
In the build system, this means python unittests will be treated the same as all
other test suites that use manifestparser. New manifests called 'python.ini' have
been created for all test directories containing python unittests.
MozReview-Commit-ID: IBHG7Thif2D
--HG--
extra : rebase_source : 11a92a2bc544d067946bbd774975140147458caa
The mozbase unittests don't use mozunit, so their output is confusing in the log.
This makes mozbase output consistent with the rest of the python unittests.
MozReview-Commit-ID: AIs5mza8Rn6
--HG--
extra : rebase_source : 10f65e612f5b3cebb921c47699f5a8be7cd2ba5a
The mozsystemmonitor test is under mozsystemmonitor/mozsystemmonitor/test instead of
mozsystemmonitor/tests like all the other mozbase modules.
MozReview-Commit-ID: AIs5mza8Rn6
--HG--
rename : testing/mozbase/mozsystemmonitor/mozsystemmonitor/test/test_resource_monitor.py => testing/mozbase/mozsystemmonitor/tests/test_resource_monitor.py
extra : rebase_source : dde714fb9212f19d1f8ba566f574bd7e9d7c4030
For some reason calling os.getpgid(proc.pid) in this bug results in an OSError "No such process"
on OSX. This bug is starting the ProcessHandler from a concurrent.futures Thread, that must be
somehow related.
I tried debugging this, but couldn't figure out why this is happening. However, the pgid is not
needed for this use case, and simply ignoring the error works. We also ignore this very same
exception when calling os.getpgid elsewhere in mozprocess, so there must be some weird OSXism
happening.
MozReview-Commit-ID: 2YXhBaORC5s
--HG--
extra : rebase_source : 120e4bff7ef29d2a0ad1e3bdd2df11b8b682d981
Adapt check_for_crashes() for latest mozcrash changes, which returns the number of crashes for both log_crashes()
and check_for_crashes() now. Also the runner should accumulate the number of crashes so it will be known at any
time how many times the process has been crashed.
MozReview-Commit-ID: Dl9FlH1TalH
--HG--
extra : rebase_source : b27895482fcad099cf4fcfc01a65fe0fbc5243e3
Currently check_for_crashes() behaves differently compared to log_crashes(), whereby it only returns a
boolean if a crash has been detected but not the amount of crash reports found. We should make sure that
both methods behave the same. Given that this change might affect consumers, we should have a major version
bump for the new release.
MozReview-Commit-ID: LiPaozJL5NF
--HG--
extra : rebase_source : d4392207399a1383a20e037bcf73f44bf3c36c7d
This causes consumers managing defaults themselves to fail to find a default
subsuite for tests, because the manifestparser will have provided a blank
default value by the time they incorporate defaults into a test definition.
This patch removes the provided defaults and updates a number of places assuming
the 'subsuite' field is always present.
MozReview-Commit-ID: 1jPy52VmEPr
This causes consumers managing defaults themselves to fail to find a default
subsuite for tests, because the manifestparser will have provided a blank
default value by the time they incorporate defaults into a test definition.
This patch removes the provided defaults and updates a number of places assuming
the 'subsuite' field is always present.
MozReview-Commit-ID: 1jPy52VmEPr
--HG--
extra : rebase_source : be7a2504af6853abb1bc532a058738f33d8dcbee
added testing/mozbase to tools/lint/flake8.lint
fixed a first batch of PEP8 errors/warnings
at first the commad autopep8 -i --max-line-length 99 -r -j 8 .
has been used to fix simpler problems, run from testing/mozbase
some of the issues can not easily fixed :
- undefined 'names' in code for example isLinux - isLinux and isBsd "fixed" with # noqa
- undefined 'message' resolved with return fmt.format(...
- undefined 'structured' resolved replacing those with mozlog
- long comments - some remaining - addressed with # noqa
- package level import everything - addressed with # flake8: noqa
restored testing/mozbase/mozdevice/mozdevice/Zeroconf.py
fixed issues reported on mozreview
fixed ')' in testing/mozbase/mozprocess/mozprocess/qijo.py imports
finally fixed multiline string at testing/mozbase/manifestparser/tests/test_manifestparser.py:114
^^^ and again, but now with ./mach python-test --path-only testing/mozbase/manifestparser/tests/test_manifestparser.py passing
fixed testing/mozbase/manifestparser/tests/test_convert_directory.py assert
fixed this error:
10:15:21 INFO - return lambda line: stack_fixer_module.fixSymbols(line)
10:15:21 INFO - TypeError: fixSymbols() takes exactly 2 arguments (1 given)
fixed two spaces lint error even of # noqa comments
restored assignement to lambda with # noqa to silence the lint error
global noqa for testing/mozbase/manifestparser/tests/test_filters.py
stupid is/is not error...
MozReview-Commit-ID: 1FpJF54GqIi
--HG--
extra : rebase_source : 3cf0277fb36a296e3506aeacc2ff05e1b03f9eac
As part of converting WPT to run from a source checkout in automation,
I ran into problems where mozinfo.json wasn't being loaded, causing
several tests to fail. This is because mozinfo.json is only found
from a source directory or from a parent of the WPT meta directory.
Before, mozinfo.json was provided by the "common" tests zip file
and extracted to the base of the destination directory. WPT files
were extracted to a "web-platform" sub-directory. So mozinfo.json
was in a parent directory of "web-platform/meta" and was loaded
automatically.
Now that we want to run WPT out of the source directory, there is
no mozinfo.json and the source directory and no mozinfo.json can be
loaded.
This commit teaches mozinfo.json to look in a MOZINFO_PATH environment
variable for the path to mozinfo.json. This allows mozinfo.json
to exist anywhere on the filesystem. This provides a lot more
flexibility for loading the file.
MozReview-Commit-ID: 1I1cxS1IeyY
--HG--
extra : rebase_source : b06de2b53d3fb26df2c7b1c6d034644c20e47577
This allows the logs to work with the structured reftest viewer.
MozReview-Commit-ID: CY71vSdDjLP
--HG--
extra : rebase_source : 6b83d98aff1c5e73ac0a802b5a83b8be95adf59a
As part of converting WPT to run from a source checkout in automation,
I ran into problems where mozinfo.json wasn't being loaded, causing
several tests to fail. This is because mozinfo.json is only found
from a source directory or from a parent of the WPT meta directory.
Before, mozinfo.json was provided by the "common" tests zip file
and extracted to the base of the destination directory. WPT files
were extracted to a "web-platform" sub-directory. So mozinfo.json
was in a parent directory of "web-platform/meta" and was loaded
automatically.
Now that we want to run WPT out of the source directory, there is
no mozinfo.json and the source directory and no mozinfo.json can be
loaded.
This commit teaches mozinfo.json to look in a MOZINFO_PATH environment
variable for the path to mozinfo.json. This allows mozinfo.json
to exist anywhere on the filesystem. This provides a lot more
flexibility for loading the file.
MozReview-Commit-ID: 1I1cxS1IeyY
--HG--
extra : rebase_source : 6fb60257b6a20f6bd79461f1bf1f4f5be1f28072
Add the `pytest_mozlog` module (from https://github.com/davehunt/pytest-mozlog)
to mozbase/mozlog/mozlog, so that we can use it in the logging of the
Marionette harness tests.
Add an entry point for the plugin to mozlog/setup.py so that
when mozlog is installed, pytest will register the pytest_mozlog
plugin.py module as the plugin 'mozlog'.
Make the following modifications to pytest_mozlog/plugin.py:
- Disable stdout logging by default in pytest-mozlog
In the pytest-mozlog plugin, disable logging to stdout by
default when setting up the mozlog logger, so that the
default output seen when using the plugin is just the
Pytest terminal output. The user can specify a mozlog
logger to write to stdout by using the '--log-* -' option,
in which case both Pytest and pytest-mozlog will log to stdout.
- Record error msg and lineno in pytest-mozlog logs
Extract the error message and line number for tests with failures/
errors from the Pytest test report, and pass these to the Mozlog
logs output by the pytest-mozlog plugin. For skipped tests,
pass the skip reason, and for unexpected passes allow Mozlog to
report its default message "expected FAIL".
- Report collected item count in pytest-mozlog
Add/modify Pytest hooks in the pytest-mozlog plugin so that the
suite_start event sent to the logger contains the correct list
of tests to be run, i.e. items collected by Pytest, instead
of an empty list. This allows mozlog to log the correct number
of tests to be run.
- Log skipped/xfailed tests with corresponding status (not 'PASS')
Log skipped tests with status 'SKIP' and expected failures
that do indeed fail with status 'FAIL'/expected 'FAIL'
so that these can be distinguished from true passing tests.
- Pass parsed arguments directly to mozlog.commandline.setup_logging
Pass the entire Pytest config's argument namespace to `setup_logging`
instead of copying over only the non-None formatters/formatter option
arguments. Use the new `allow_unused_options` parameter to stop
Mozlog from raising an error about options for unused formatters.
MozReview-Commit-ID: 4WpprytS5cI
--HG--
extra : rebase_source : 409de7a89e860471870e3adfd7634bc06f6e1873
Add a parameter `allow_unused_options` to the `setup_logging` and
`setup_handlers` methods in mozlog.commandline. When this parameter
is True, do not raise a ValueError in `setup_handlers` if the
command line arguments include formatter options for unused formatters.
(Default value is False.)
This allows parsed arguments from e.g. the pytest-mozlog plugin
to be passed directly to `setup_logging`, without filtering out
unused formatter options.
MozReview-Commit-ID: 8el0oaT4y1X
--HG--
extra : rebase_source : 0e969a783b1754885e6c25d35c552134773f1b5e
In mozdevice, it's possible to pass in the path to the adb binary (rather than requiring it to be on the path). The
mochitest android options don't provide any means to set this however (even though the b2g options do). This patch
adds that option in.
This will be used by the mach environment on interactive loaners.
MozReview-Commit-ID: 4lsKGpizfH7
--HG--
extra : rebase_source : 07ad43c8fff3eb83a1eaab9ad0532cd85c4532c7
When running the command for starting Fennec, quotation marks aren't
processed properly when the 'am' portion of the command is represented
with one string token per argument; the args must be joined into one string
instead.
Also add log message about command being run in BaseRunner.
MozReview-Commit-ID: KZLnOdu9UGq
--HG--
extra : rebase_source : c8072c003afec3aba18f307b8cc64332736e55fc
This fixes an error when attempting to run xpcshell-test with a test path from an interactive loaner.
MozReview-Commit-ID: 20kg5zKplhT
--HG--
extra : rebase_source : cda8e17dd6ad7b606711630e90df8d5e65c8bda3
Ensure that the first line of the log for failing tests is identical to
one that would be produced by the tbplformatter from mozlog. This means
that treeherder will be able to sucessfully cross-match error lines.
Also make the data structures used for storing screenshots identical
between reftests and web-platform-tests so that we can start to move
toward removing the reftest-specific codepath here.
MozReview-Commit-ID: FZQXLjj9Ejv
--HG--
extra : rebase_source : 3e9bf70ff7915817b8c02032c8da0d0a659e4661
This fixes a regression from bug 1288827. It happened because I moved the logic that finds
the application path a little later on in the test harness. But there was an instance where
it was being used in the android mach command before that point.
As it turned out, we don't really *need* that value there. This patch grabs the same value
from build_obj.substs which is already an argument to the function.
MozReview-Commit-ID: 3IsI4VzEIIF
--HG--
extra : rebase_source : ceb718b265c7886b831e0378436baebf2b4e7598
This removes the unnecessary setting of c-basic-offset from all
python-mode files.
This was automatically generated using
perl -pi -e 's/; *c-basic-offset: *[0-9]+//'
... on the affected files.
The bulk of these files are moz.build files but there a few others as
well.
MozReview-Commit-ID: 2pPf3DEiZqx
--HG--
extra : rebase_source : 0a7dcac80b924174a2c429b093791148ea6ac204
Any exception which gets thrown by a log handler while test results are getting generated, should
not cause test harnesses to stop immediately. To achive that the exception details are written to
stderr and not propagated up the stack.
MozReview-Commit-ID: ChyYxApYSGx
--HG--
extra : rebase_source : 9fc3fe597061bedb1df2f5b8de1daa4bd127ea1e
There are two identical pairs of screenshots (test/reference pair) which are
png/base64 raw files generated from mozlog's HTML formatter. One pair is stored
in the img element to present the visual result; the other pair is stored as a
hyperlink source in the visual result's title.
After part1 patch, we may have one more pair. It appears that the hyperlinks of
the visual result's titles could be eliminated since they are visually closed to
the visual results, and clicking the visual results provides the exact same
function.
DONTBUILD (NPOTB)
MozReview-Commit-ID: 4CLfYXX8g69
--HG--
extra : rebase_source : d4d3b4e08b66e737d75a2ca21b6e84b344a29fc8
Current mozlog (v3.2) doesn't support screenshot logs exported from wptrunner.
Add this support so we could run css test with --log-html to see more detail
information, such as screenshots of test/reference pages.
DONTBUILD (NPOTB)
MozReview-Commit-ID: AUJwYfvNfda
--HG--
extra : rebase_source : 1d3d1fcdc396638d256336d3dfaf5ba5bad35168
This is a refactor. |start_logcat| allows filtering by tag and is used for
recording gecko.log for Marionette tests on Fennec.
MozReview-Commit-ID: 9NO6jQDMQ9E
--HG--
extra : rebase_source : e4f60b5d1c9c4ed6bb6dd237d9c1433b5f04a8d6
This works around issues with System Integrity Protection kicking in on OSX when
trying to run /usr/bin/lldb by attempting to using the version installed with
XCode's command line utilities.
In particular SIP prevents us from setting DYLD_LIBRARY_PATH which is needed
to run DMD.
Add FennecEmulatorRunner (for convenience), FennecRunner, FennecContext
and EmulatorAVD.
Common behaviour is defined in BaseEmulator and RemoteContext to distinguish
from B2G and Fennec specifics. I've tried to decouple ArchContext from
B2GContext, as well.
The emulator/adb commands in FennecRunner and EmulatorAVD are intended to
match the behaviour seen in current Android automation (e.g. mochitest).
MozReview-Commit-ID: 1tqD0DStdHR
--HG--
extra : rebase_source : 1450f3b03f82a0f9d33e43d19632a06a51ef7253
MozReview-Commit-ID: 9e6HNAaF0Yo
In case of in-process restarts it can happen that the new process gets forked into a new process group.
When that happens we loose the capability to kill the process. To prevent a hang when joining the output
reader threads in wait(), we simply skip that call by passing-through the IO error.
--HG--
extra : rebase_source : 702dfec407ed13114f59fa6ccb0d82c5b0790550
Sometimes the IO completion port doesn't shutdown child processes. When this happens,
mozprocess will attempt to force kill the child processes manually. However, there is
a bug here which causes the OSError to get raised.
Although this fixes that bug, the original issue(s) which prevented the IOC port
from signaling shutdown remain and are still undiagnosed.
MozReview-Commit-ID: L3DQPW0Is5v
--HG--
extra : rebase_source : cf6320cffea5a4c8fb5d62861c41065d9dcefa52
We can set MOZPROCESS_DEBUG to help debug windows process code. However on try
it is unreadable as there are multiple things using mozprocess, and each process
has multiple threads. It's impossible to tell which log message comes from where.
This improves the debug logs a bit by always specifying the PID and thread name.
There are a few other drive-by cleanups in this thread. The only one of note is
removing a python 2.5 only code path.
MozReview-Commit-ID: L3DQPW0Is5v
--HG--
extra : rebase_source : f07c07f53b06b1160cd3e70cb06b8dc12a89c3ab
This code is no longer used by Marionette client or elsewhere.
MozReview-Commit-ID: 4lx9CN7XIeH
--HG--
extra : rebase_source : e0a895c02939c51ee40be5be5f999cc41420a2a7
I believe this is the source of hangs/timeouts in automation.
join() waits forever. We add code to wait at most N seconds before
force terminating the process. The timeout is a bit high. But it is
better than infinite.
MozReview-Commit-ID: KwyO4RZ9OqL
--HG--
extra : rebase_source : 767d8ff5b48d7e75ab8fe72b18145446a38d439a
Before, we kept waiting for data in the pipe after receiving the
"done" message. This didn't really make much sense because the
"done" message should be the final thing sent over the pipe!
e9113fd6cdb8 (bug 1239939) recently dropped the poll interval
of the pipe from 1.0 to 0.1s. This appears to have introduced
an intermittent failure in a test. The race condition was
between the child process sending data and the parent process
timing out (after only 0.1s) waiting for that data. Increasing
the timeout makes the failure reproduce less often. Although
technically the race condition is still present! I'm not
inclined to fix it at this time, however.
The rationale for dropping the pipe timeout was that it was
causing lag when terminating short-lived processes. Now that
we abort the pipe reading/polling loop as soon as the "done"
message is received, we no longer poll the pipe after receiving
"done" and no longer have to worry about its timeout impacting
shutdown time.
MozReview-Commit-ID: EeENQ95RAs1
--HG--
extra : rebase_source : ce2502f32841a55f912aafdba7cc81e3a58e3ff5
Found this bug when auditing the code for issues. We are attempting
to send a tuple but were forgetting the trailing comma on a single
element tuple.
Fortunately, this doesn't appear to impact anything because
the receiving end of the pipe doesn't care what data it receives.
MozReview-Commit-ID: E34fBqxgUSq
--HG--
extra : rebase_source : 3701a28979a8b53d40ea68acef3ee2cb6d8a50f2
We add some system information including processor count
and memory sizes. We also add an "overall" section describing
total resource usage. This (surprisingly) wasn't defined.
This commit is the first in a series to reconcile the differences
between the JSON format in mozsystemmonitor and what
`mach build` writes so we can write a single tool to visualize
the data.
MozReview-Commit-ID: 9mdbKxeV9Ta
--HG--
extra : rebase_source : 3aadf5e83c91ba9553595f3da77ed7ca0e4d5541
We're currently running version 0.0 in automation. This version
doesn't have as_dict(), which means we can't easily save data
to JSON.
Bump the version to 0.1 in preparation of releasing a new
version.
MozReview-Commit-ID: Kr3JqyRXk5j
--HG--
extra : rebase_source : 89f763acaa12e4357f4a23f8772f99c1a0fdb56f
We have packages for 3.1.1 uploaded to our PyPI server used
for automation. There have been a number of bug fixes since the
version of psutil currently listed. Let's ensure we're running
a modern psutil to minimize our exposure to bugs on older
versions.
MozReview-Commit-ID: 6rDapZ8miFD
--HG--
extra : rebase_source : c66295828e0c95c4ffe57e579df41af508875027
This requires a change to how we process test manifests in the build system:
now, whenever we see a support file mentioned in a manifest, we require that
file isn't already in that test's support files, but if we see a support file
that was already seen in some other test, the entry is ignored, but it is not
an error. As a result of this change, several duplicate support-files entries
needed to be removed.
MozReview-Commit-ID: G0juyxzcaB8
--HG--
rename : testing/mozbase/manifestparser/tests/test_default_skipif.py => testing/mozbase/manifestparser/tests/test_default_overrides.py
Structured logs bring many benefits. We can stop parsing the logs for magic strings, we
can modify the format without breaking things, and we can stream results into systems like
ActiveData. The structured logs originate primarily in reftest.js. StructuredLog.jsm is
used to generate the JSON-based log stream. Finally OutputHandler in the python harness
reads structured output from stdout, and formats it into human readable form.
MozReview-Commit-ID: G3ZLkMRl6p7
--HG--
extra : commitid : J3ui9XYWR3Q
extra : rebase_source : 77ed0ba842cc8e557141fb3494212b06868c728a
extra : amend_source : 735d48225a2e627e0fe45fc11b50b6c49a885a4b
extra : source : d1779fe421c3c7cd8e3d191816776390dc104f37
This approach lets us get rid of the .App <activity-alias> definitions, at
least.
In Bug 1237755, I tried pretty hard to use real intent filters, rather
than a different hard-code path name, but ran into significant issues,
and finally into the hard reality that Android 2.3 doesn't support
bare package names. (It only supports package/class component names.)
Android 2.3 can't die fast enough.
Sadly, this approach doesn't address the issues I see loading URLs
that prompted Bug 1237755.
This is the path of least resistance. Onwards!
Tested via:
nalexander@chocho ~/M/gecko> ./mach run
Starting: Intent { act=android.activity.MAIN cmp=org.mozilla.fennec_nalexander/org.mozilla.gecko.BrowserApp }
Warning: Activity not started, its current task has been brought to the front
nalexander@chocho ~/M/gecko> ./mach run https://mozilla.org
Starting: Intent { act=android.activity.MAIN dat=https://mozilla.org cmp=org.mozilla.fennec_nalexander/org.mozilla.gecko.BrowserApp }
nalexander@chocho ~/M/gecko> ./mach run file:///mnt/sdcard/test.html
Starting: Intent { act=android.activity.MAIN dat=file:///mnt/sdcard/test.html cmp=org.mozilla.fennec_nalexander/org.mozilla.gecko.BrowserApp }
--HG--
extra : commitid : JUIJj13Q0Py
extra : rebase_source : a62d8a74ca2772fd3933597589e386cf56caa396
extra : amend_source : 234f3b6b8ef6c6f2cce991375f5c28fe1cb32cad
extra : histedit_source : 5476222279f6be0927879a1136def94673885071%2Cca4aad42479731c97586003ee8bcdfe22a147776
Currently we poll at an interval of 1 second when shutting down
mozsystemmonitor. This causes a noticeable shutdown delay on short
running processes. Reducing the interval to 0.1 reduces this delay without
seeming to have any adverse effects.
--HG--
extra : rebase_source : da59ee8d09ada188d011e3a7138b30d91850aacd
This now check process status based on pids, not anymore
on an arbitrary string given in the command line.
Some cleanup in there, including removing code duplicate.
We need to go through an abstraction layer when manifestparser manifests
are read as a part of reading moz.build files in case moz.build reading
is using mercurial as its filesystem. This adds an optional finder member,
which will be used if present for IO that happens when reading manifests,
This needs to be optional, because the manifestparser can be used as a
standalone package that isn't distributed with mozpack.
--HG--
extra : commitid : 7S8mdpexvgW
Profiling the manifestparser revealed that 25% of its total time is spent in a single
call to os.path.normpath, called on each test path in each manifest. For the manifests
in mozilla-central, these calls almost always return their input. To save time, this
patch modifies the manifestparser to not call normpath on paths that don't contain
'..'. This will change the parsed result of manifests containing no-op path
components ('//' or '/./') to include those components in their output.
--HG--
extra : commitid : D7g4GX6VLuC
This prevents copying of 447 files adding to ~4 MB.
--HG--
extra : commitid : 7zTbiQeMQSQ
extra : rebase_source : b3ac223835ba7289ace45aa7d02c5a050d54cc0d
This stops exposing ANDROID_BUILD_TOOLS and ANDROID_PLATFORM_TOOLS via
AC_SUBST. We expose most tools already, and this adds EMULATOR, and
consumes it (and ADB) where appropriate.
--HG--
extra : commitid : 9u0pibgE00
extra : rebase_source : 04e420c53d1d75ab8f055436d7dd69e148168c67
extra : histedit_source : a930a34f4dda44ee91b52caf68e02877b0502f01
There is an assumption in application.py that there exists a build with a valid bindir. This directory
doesn't exist if no build exist. Before now, this code path hasn't been a problem as it was only used
with built emulators. But now gaia is starting to use it and there might not even be a B2G repo at all!
--HG--
extra : commitid : 3sxXkmDErLG
extra : rebase_source : 526d89f36371970f8b7edca20e3dbb01131f90c7
The install.rdf in language packs uses em:id attributes etc, instead
of child elements. All fine with RDF/XML.
This adds support to addon_details to allow for those attributes in
addition to child elements.
--HG--
extra : transplant_source : %E0d%E7%B3%E8/%BF%D4%5C%AE%B8EA%E7%26h%17%5EB%00
For contemporary versions of Windows the sys.getwindowsversion call is unreliable
because it depends on a API which had its return value frozen. If we detect this
case use an alternate API instead
For contemporary versions of Windows the sys.getwindowsversion call is unreliable
because it depends on a API which had its return value frozen. If we detect this
case use an alternate API instead
The manifestparser dependency of mozprofile is made optionnal with this patch.
It is still possible to install it using the extra pip/setuptools syntax:
pip install mozprofile[manifest]
--HG--
extra : commitid : GN6pzPlCHgF
extra : rebase_source : 9d258961c8fc3e8c314e9f2f7d59e0f6687e6640
This also completely remove build/automationutils.py.
--HG--
extra : commitid : 50v6EAQNEHV
extra : rebase_source : 4ac1347d73498f068979514c6afb16ac50ab4033
Tarek Ziadé
Sylvestre Ledru
Alex Gaynor
Sebastian Hengst
Andrew Halberstadt
Christian Holler
James Graham
Carsten "Tomcat" Book
William Lachance
Dão Gottwald
Geoff Brown
Ting-Yu Chou
Maja Frydrychowicz
Wes Kocher
Andrea Marchesini
Iris Hsiao
James Graham
Sam Giles
Dave Hunt
Jim Chen
Bob Clary
Florian Quèze
Brad Lassey
John Paul Adrian Glaubitz
Joel Maher
Henrik Skupin
Andreas Tolfsen
Julian Seward
Kris Maglione
Ehsan Akhgari
Chris Manchester
Ryan VanderMeulen
Randell Jesup
Gregory Szorc
Ted Mielczarek
David Anderson
Kartikaya Gupta
Francesco Pischedda
Phil Ringnalda
Anjana Vakil
Towkir Ahmed
Andrew McCreight
Tom Tromey
Rob Thijssen
Jeremy Chen
Eric Rahm
Nicolas B. Pierron
Xidorn Quan
James Willcox
Shing Lyu
Ms2ger
Richard Newman
Jeff Walden
Julien Pagès
Nick Alexander
Alessio Placitelli
Dan Minor
Nathan Froyd
Armen Zambrano Gasparnian
Edgar Chen
Steve Fink
Tooru Fujisawa
dlee
Jose Diez
Axel Hecht
Hiroyuki Ikezoe
Rahul Sharma
Michael Layzell