This is a new issue that gets linted with flake8 3.5.0. Basically you should
never use a blank except: statement.
This will catch all exceptions, including KeyboardInterrupt and SystemExit
(which is likely not intended). If a catch all is needed, use
`except: Exception`. If you *really* mean to also catch KeyboardInterrupt et
al, use `except: BaseException`.
Of course, being specific is often better than a catch all.
MozReview-Commit-ID: FKx80MLO4RN
--HG--
extra : rebase_source : 7c74a7d0d81f2c984b47aff3a0ee3448b791177b
The poll() call in SystemResourceMonitor.stop might fail even though
there is something to read from the pipe, in some corner cases, and
python won't let us know about it. In that case, an exception is thrown,
leaving the SystemResourceMonitor (and its callers) in a weird state. In
practice, this leads BuildMonitor.__exit__ to recall stop, which then
fails.
So when poll() throws an exception, we pretend there's still something
to read, and we try to read anyways. If there is something to read,
recv() will return it, otherwise, it will throw an exception of its own,
which we catch, pretending we're done.
Furthermore, when there is nothing to read from the pipe, poll() simply
returns False, and our loop never sets `done` to True, and we then hit
an assert, which doesn't have its place here, so we remove it.
Finally, the other end of the pipe might have died at any time, making
sending over the pipe fail, so we also protect against that.
With all these changes, it feels like the reason to backout bug 1239939
in bug 1272782 should have been dealt with, and we can drop the timeout
again.
--HG--
extra : rebase_source : ac72dd5b2602cf3ffddfb429f95e02380f939893
The poll() call in SystemResourceMonitor.stop might fail even though
there is something to read from the pipe, in some corner cases, and
python won't let us know about it. In that case, an exception is thrown,
leaving the SystemResourceMonitor (and its callers) in a weird state. In
practice, this leads BuildMonitor.__exit__ to recall stop, which then
fails.
So when poll() throws an exception, we pretend there's still something
to read, and we try to read anyways. If there is something to read,
recv() will return it, otherwise, it will throw an exception of its own,
which we catch, pretending we're done.
Furthermore, when there is nothing to read from the pipe, poll() simply
returns False, and our loop never sets `done` to True, and we then hit
an assert, which doesn't have its place here, so we remove it.
Finally, the other end of the pipe might have died at any time, making
sending over the pipe fail, so we also protect against that.
With all these changes, it feels like the reason to backout bug 1239939
in bug 1272782 should have been dealt with, and we can drop the timeout
again.
--HG--
extra : rebase_source : fededf989fe9021654b67d5a070f7e49aa717f3c
The subsuite is added conditionally because we only have the capability of
running source-check tasks on linux at the moment. Once taskcluster support
for windows and mac has matured a bit and the taskcluster configs support
source-check there, we should apply the subuite unconditionally.
MozReview-Commit-ID: Kk9Irz3fn14
--HG--
extra : rebase_source : b9266a06583083c36477d4e93f5462ee614cdb71
The mozbase unittests don't use mozunit, so their output is confusing in the log.
This makes mozbase output consistent with the rest of the python unittests.
MozReview-Commit-ID: AIs5mza8Rn6
--HG--
extra : rebase_source : 10f65e612f5b3cebb921c47699f5a8be7cd2ba5a
The mozsystemmonitor test is under mozsystemmonitor/mozsystemmonitor/test instead of
mozsystemmonitor/tests like all the other mozbase modules.
MozReview-Commit-ID: AIs5mza8Rn6
--HG--
rename : testing/mozbase/mozsystemmonitor/mozsystemmonitor/test/test_resource_monitor.py => testing/mozbase/mozsystemmonitor/tests/test_resource_monitor.py
extra : rebase_source : dde714fb9212f19d1f8ba566f574bd7e9d7c4030
added testing/mozbase to tools/lint/flake8.lint
fixed a first batch of PEP8 errors/warnings
at first the commad autopep8 -i --max-line-length 99 -r -j 8 .
has been used to fix simpler problems, run from testing/mozbase
some of the issues can not easily fixed :
- undefined 'names' in code for example isLinux - isLinux and isBsd "fixed" with # noqa
- undefined 'message' resolved with return fmt.format(...
- undefined 'structured' resolved replacing those with mozlog
- long comments - some remaining - addressed with # noqa
- package level import everything - addressed with # flake8: noqa
restored testing/mozbase/mozdevice/mozdevice/Zeroconf.py
fixed issues reported on mozreview
fixed ')' in testing/mozbase/mozprocess/mozprocess/qijo.py imports
finally fixed multiline string at testing/mozbase/manifestparser/tests/test_manifestparser.py:114
^^^ and again, but now with ./mach python-test --path-only testing/mozbase/manifestparser/tests/test_manifestparser.py passing
fixed testing/mozbase/manifestparser/tests/test_convert_directory.py assert
fixed this error:
10:15:21 INFO - return lambda line: stack_fixer_module.fixSymbols(line)
10:15:21 INFO - TypeError: fixSymbols() takes exactly 2 arguments (1 given)
fixed two spaces lint error even of # noqa comments
restored assignement to lambda with # noqa to silence the lint error
global noqa for testing/mozbase/manifestparser/tests/test_filters.py
stupid is/is not error...
MozReview-Commit-ID: 1FpJF54GqIi
--HG--
extra : rebase_source : 3cf0277fb36a296e3506aeacc2ff05e1b03f9eac
I believe this is the source of hangs/timeouts in automation.
join() waits forever. We add code to wait at most N seconds before
force terminating the process. The timeout is a bit high. But it is
better than infinite.
MozReview-Commit-ID: KwyO4RZ9OqL
--HG--
extra : rebase_source : 767d8ff5b48d7e75ab8fe72b18145446a38d439a
Before, we kept waiting for data in the pipe after receiving the
"done" message. This didn't really make much sense because the
"done" message should be the final thing sent over the pipe!
e9113fd6cdb8 (bug 1239939) recently dropped the poll interval
of the pipe from 1.0 to 0.1s. This appears to have introduced
an intermittent failure in a test. The race condition was
between the child process sending data and the parent process
timing out (after only 0.1s) waiting for that data. Increasing
the timeout makes the failure reproduce less often. Although
technically the race condition is still present! I'm not
inclined to fix it at this time, however.
The rationale for dropping the pipe timeout was that it was
causing lag when terminating short-lived processes. Now that
we abort the pipe reading/polling loop as soon as the "done"
message is received, we no longer poll the pipe after receiving
"done" and no longer have to worry about its timeout impacting
shutdown time.
MozReview-Commit-ID: EeENQ95RAs1
--HG--
extra : rebase_source : ce2502f32841a55f912aafdba7cc81e3a58e3ff5
Found this bug when auditing the code for issues. We are attempting
to send a tuple but were forgetting the trailing comma on a single
element tuple.
Fortunately, this doesn't appear to impact anything because
the receiving end of the pipe doesn't care what data it receives.
MozReview-Commit-ID: E34fBqxgUSq
--HG--
extra : rebase_source : 3701a28979a8b53d40ea68acef3ee2cb6d8a50f2
We add some system information including processor count
and memory sizes. We also add an "overall" section describing
total resource usage. This (surprisingly) wasn't defined.
This commit is the first in a series to reconcile the differences
between the JSON format in mozsystemmonitor and what
`mach build` writes so we can write a single tool to visualize
the data.
MozReview-Commit-ID: 9mdbKxeV9Ta
--HG--
extra : rebase_source : 3aadf5e83c91ba9553595f3da77ed7ca0e4d5541
We're currently running version 0.0 in automation. This version
doesn't have as_dict(), which means we can't easily save data
to JSON.
Bump the version to 0.1 in preparation of releasing a new
version.
MozReview-Commit-ID: Kr3JqyRXk5j
--HG--
extra : rebase_source : 89f763acaa12e4357f4a23f8772f99c1a0fdb56f
We have packages for 3.1.1 uploaded to our PyPI server used
for automation. There have been a number of bug fixes since the
version of psutil currently listed. Let's ensure we're running
a modern psutil to minimize our exposure to bugs on older
versions.
MozReview-Commit-ID: 6rDapZ8miFD
--HG--
extra : rebase_source : c66295828e0c95c4ffe57e579df41af508875027
Currently we poll at an interval of 1 second when shutting down
mozsystemmonitor. This causes a noticeable shutdown delay on short
running processes. Reducing the interval to 0.1 reduces this delay without
seeming to have any adverse effects.
--HG--
extra : rebase_source : da59ee8d09ada188d011e3a7138b30d91850aacd
The build system / mach currently has a very hacky virtualenv setup.
Essentially, it resorts to sys.path munging instead of a proper,
isolated environment.
During initialization, mach installs python/psutil in sys.path. Later
on, some code does an |import psutil|. This fails iff the psutil C
extension can't be found.
If there is a psutil C extension installed outside of mach and
python/psutil, |import psutil| may load it. The version mismatch isn't
detected until an extension-using psutil API is called. This has
manifested inside |mach build| via the resource monitor as an
|AttributeError: 'module' object has no attribute 'linux_sysinfo'|
exception during psutil.virtual_memory().
The proper fix for this is for the Python environment to ensure the
psutil C extension is built before attempting to import and use psutil.
Arguably, psutil itself should perform some kind of version check when
it imports the C extension to ensure things are in sync and fail at
import time.
Fixing mach and the build system Python environment to build psutil
earlier/properly is a long outstanding bug. It needs to be addressed.
But it is considerable effort. This patch continues the long history of
wallpapering over psutil import/run failures because using a proper
virutalenv from mach/build system is a lot of work. Sad panda.
--HG--
extra : rebase_source : 5c449d69c0fd907ea8359ac721ef6287baa4f10e
Gregory Szorc
Andrew Halberstadt
Tom Prince
Steve Armand
Mike Hommey
Phil Ringnalda
Jim Chen
Francesco Pischedda
Dan Minor
Mike Shal
Chris Manchester