Allow-list all Python code in tree for use with the black linter, and re-format all code in-tree accordingly.
To produce this patch I did all of the following:
1. Make changes to tools/lint/black.yml to remove include: stanza and update list of source extensions.
2. Run ./mach lint --linter black --fix
3. Make some ad-hoc manual updates to python/mozbuild/mozbuild/test/configure/test_configure.py -- it has some hard-coded line numbers that the reformat breaks.
4. Make some ad-hoc manual updates to `testing/marionette/client/setup.py`, `testing/marionette/harness/setup.py`, and `testing/firefox-ui/harness/setup.py`, which have hard-coded regexes that break after the reformat.
5. Add a set of exclusions to black.yml. These will be deleted in a follow-up bug (1672023).
# ignore-this-changeset
Differential Revision: https://phabricator.services.mozilla.com/D94045
Allow-list all Python code in tree for use with the black linter, and re-format all code in-tree accordingly.
To produce this patch I did all of the following:
1. Make changes to tools/lint/black.yml to remove include: stanza and update list of source extensions.
2. Run ./mach lint --linter black --fix
3. Make some ad-hoc manual updates to python/mozbuild/mozbuild/test/configure/test_configure.py -- it has some hard-coded line numbers that the reformat breaks.
4. Add a set of exclusions to black.yml. These will be deleted in a follow-up bug (1672023).
# ignore-this-changeset
Differential Revision: https://phabricator.services.mozilla.com/D94045
Use universal_newlines as a convenient way of treating the process I/O as text,
so string read/writes continue to succeed under python 3.
Differential Revision: https://phabricator.services.mozilla.com/D81774
Hopefully this will avoid some intermittent failures that occur when
`fix-stacks` is deleted at the end of a test run.
Differential Revision: https://phabricator.services.mozilla.com/D76980
Because the web platform test harness doesn't recognize them and annotates them
with `CRITICAL`, which is disconcerting, even though those errors are typically
not harmful.
Differential Revision: https://phabricator.services.mozilla.com/D71485
Currently AWSY-with-DMD doesn't work on Windows. This is because `fix-stacks`
is initialized lazily, and by the time the initialization happens some file
descriptors for files are open, and that leads to some major Python2-on-Windows
sadness as described in the big comment in the commit.
To fix the problem, this commit adds an `init` function to `fix_stacks.py` so
that `fix-stacks` can be initialized eagerly, hopefully before any file
descriptors for files are open.
For `dmd.py`, other than fixing the AWSY problems, this has little effect,
because `fix-stacks` is always initialized.
For `utils.py`, which is used to process the output of most tests, this has a
more noticeable effect: the `fix-stacks` process is always spawned, rather than
being spawned only when needed. If no stack traces appear in the test output,
this means that `fix-stacks` is spawned unnecessarily. But it's cheap to spawn;
the expensive part only happens when stack traces start getting fixed. So I
think this change in behaviour is acceptable.
Furthermore, the commit adds a `finish` function to `fix_stacks.py`, so that
the `fix-stacks` process can be explicitly shut down. This has never been done
for processes spawned for any of the stack fixing scripts. It's never caused
problems on Linux/Mac, but it seems to be necessary on Windows to avoid
similar "this file is locked" problems with the test_dmd.js test.
The commit also renames some things to more standard Python style, e.g.
`json_mode` instead of `jsonMode`.
Finally, Android tests use `utils.py` from the repository but `fix_stacks.py`
from the Android host utils. Because the two scripts must be updated in tandem,
this commit also updates the Android host utils to a version that contains the
updated `fix_stacks.py`. Thanks to aerickson for packaging up the new Android
host utils and providing the change to the `hostutils.manifest` file.
Differential Revision: https://phabricator.services.mozilla.com/D69478
This commit removes `test_fix_stack_using_bpsyms.py`. That test can't easily be
modified to work with `fix_stacks.py` because it relies on internal
implementation details of `fix_stack_using_bpsym.py`. The unit testing done in
the `fix-stacks` repo provides test coverage that is as good or better.
Differential Revision: https://phabricator.services.mozilla.com/D66924
--HG--
extra : moz-landing-system : lando
Currently AWSY-with-DMD doesn't work on Windows. This is because `fix-stacks`
is initialized lazily, and by the time the initialization happens some file
descriptors for files are open, and that leads to some major Python2-on-Windows
sadness as described in the big comment in the commit.
To fix the problem, this commit adds an `init` function to `fix_stacks.py` so
that `fix-stacks` can be initialized eagerly, hopefully before any file
descriptors for files are open.
For `dmd.py`, other than fixing the AWSY problems, this has little effect,
because `fix-stacks` is always initialized.
For `utils.py`, which is used to process the output of most tests, this has a
more noticeable effect: the `fix-stacks` process is always spawned, rather than
being spawned only when needed. If no stack traces appear in the test output,
this means that `fix-stacks` is spawned unnecessarily. But it's cheap to spawn;
the expensive part only happens when stack traces start getting fixed. So I
think this change in behaviour is acceptable.
The commit also renames some things to more standard Python style, e.g.
`json_mode` instead of `jsonMode`.
Differential Revision: https://phabricator.services.mozilla.com/D69478
--HG--
extra : moz-landing-system : lando
Now that `fix_stacks.py` is being used instead of `fix_macosx_stack.py`,
stack-fixing time has dropped from about 14 minutes to about 30 seconds on my
new MacBook Pro.
Also, print a warning about stacks not being fixed if `MOZ_DISABLE_STACK_FIX`
is set. This warning shows up at the start of the test run.
Also, print a warning about stack fixing slowness, because 30 seconds is long
enough to possibly be surprising. This warning shows up just before the first
stack frame is fixed, like this:
```
Assertion failure: false (BEEP BOOP), at /home/njn/moz/au3/dom/base/nsGlobalWindowOuter.cpp:1342
Initializing stack-fixing for the first stack frame, this may take a while...
#01: nsGlobalWindowOuter::nsGlobalWindowOuter(unsigned long) (/home/njn/moz/au3/dom/base/nsGlobalWindowOuter.cpp:1342)
#02: ...
```
Differential Revision: https://phabricator.services.mozilla.com/D65676
--HG--
extra : moz-landing-system : lando
Specifically:
- handle input that doesn't end in a newline;
- only pass lines that match the regexp to `fix-stacks`.
These changes are needed for `utils.py` to use `fix_stacks.py` without
problems.
Differential Revision: https://phabricator.services.mozilla.com/D65944
--HG--
extra : moz-landing-system : lando
On Linux and Mac, this makes `dmd.py` *much* faster when it is first run on a
DMD data file.
On Windows, this makes DMD actually usable locally. Previously the stacks
weren't fixed and so were rubbish.
Differential Revision: https://phabricator.services.mozilla.com/D57271
--HG--
extra : moz-landing-system : lando
So that dmd.py isn't broken by function names that, after stack fixing, contain
escape-worthy chars such as " and /.
--HG--
extra : rebase_source : 625cef56e4599bbb810273ef04af2b971e777b58
When reworking the script each entry holding a function name was replaced by a
dictionary holding both the function name and its size. This significantly
increased memory consumption as using a full-fledged dictionary for only two
fields is very space inefficient. This patch uses a named tuple instead of a
dictionary for every entry, reducing memory consumption by almost four times.
Differential Revision: https://phabricator.services.mozilla.com/D6603
--HG--
extra : moz-landing-system : lando
This patch changes the way we search symbols when fixing up a stack.
Previously we would find the closest PUBLIC or FUNC entry lower than a given
address. Because of how symbol files were processed we preferred PUBLIC
entries to FUNC ones. Now we look first for the function that contains the
address (obtained from the FUNC entries) then if none is available we look for
the closest, lower PUBLIC entry.
Differential Revision: https://phabricator.services.mozilla.com/D5033
--HG--
extra : moz-landing-system : lando
This includes both the vanilla sources we haven't forked and the client
sources that we have. Client patches were applied manually up to version
69c2c51dd89965d234eec16e3a9353634831916b. The following changes were not
included as they break merging segments corresponding to libxul.so in the
module list:
8915f7be39448d9257b6da3ad0233944d1d9a92a
17ad0c18b179c135fc5a3d2bba199c3fa4276035
94b6309aecaddfcf11672f6cfad9575d68ad3b40
With these changes applied two entries for libxul.so are generated, the second
one is bogus and prevents symbolication from working correctly.
The build system and some of the tools relying on breakpad were also updated
to work with the new version.
--HG--
extra : source : fe4d49307f8890a0c430c257c96f74a9552eeb31
extra : histedit_source : bc84861445bd93856cd0d0c864fd15ad7d9ccc12%2C1efd65797da46e33481afa61a302098780b0f107
These locations have changed since the end of Google Code.
MozReview-Commit-ID: FSGhFBDaTCq
--HG--
extra : rebase_source : 4ef004b2d0b8f1a6331db0f44ccf70d05ad62774
If the symbols file doesn't give a function name, substitute
'unnamed_function'. A more complete fix would be to catch
all ValueError exceptions and continue sensibly, but this
addresses the immediate issue with the output from `rustc -g`.
MozReview-Commit-ID: 666ruvLlJ5t
This patch introduces a small utility program to extract a guid from a shared library
or executable on windows to identify the correct symbol file to read in fix_stack_using_bpsyms.py.
In order for this to work correctly on windows, the library name provided by
MozDescribeCodeAddress needs to be a full path, so the LoadedImageName field
from the IMAGEHLP_MODULE64 structure is used here instead of the ModuleName
field.
MozReview-Commit-ID: 8zkfLWjKVs2
This patch introduces a small utility program to extract a guid from a shared library
or executable on windows to identify the correct symbol file to read in fix_stack_using_bpsyms.py.
In order for this to work correctly on windows, the library name provided by
MozDescribeCodeAddress needs to be a full path, so the LoadedImageName field
from the IMAGEHLP_MODULE64 structure is used here instead of the ModuleName
field.
--HG--
extra : commitid : GwkhBdm81g3
fix_stack_using_bpsyms.py locates a .sym file based on file name only, not uuid or full path,
which causes a failure if a duplicate leaf file name is introduced. This patch introduces a
small utility program on mac and linux to extract a breakpad guid from a shared library or
executable to identify the correct symbol file when this ambiguity occurs. A subsequent commit
implements this for windows.
--HG--
extra : commitid : 2O7REfHuDus
This is needed to avoid regressions relative to the perl version because
the perl version failed, due to a missing /, to properly examine two of
the paths, one of which often yields the same filename as the original
file (but whose CRC will fail).
I've tested that this CRC calculation succeeds in cases where it should
succeed.
The observed CRC check failures (so far) on my Ubuntu 14.04 system are
the following links:
/lib/x86_64-linux-gnu/libc.so.6 -> /lib/x86_64-linux-gnu/libc-2.19.so
/lib/x86_64-linux-gnu/libpthread.so.0 -> /lib/x86_64-linux-gnu/libpthread-2.19.so
/usr/lib/x86_64-linux-gnu/libgdk-x11-2.0.so.0 -> /usr/lib/x86_64-linux-gnu/libgdk-x11-2.0.so.0.2400.23
/usr/lib/x86_64-linux-gnu/libgtk-x11-2.0.so.0 -> /usr/lib/x86_64-linux-gnu/libgtk-x11-2.0.so.0.2400.23
which succeed on a later possibility in /usr/lib/debug/. But optimizing
to check f != file before checking the CRC wouldn't help because of the
naming differences due to symlinks, so I didn't bother. If we ever want
to optimize away the time doing the CRC checks for these failures, we
could do so by checking if the files are the same path *after* resolving
symlinks.
DONTBUILD
Bogdan Tara
Ricky Stewart
Dorel Luca
Mike Hommey
Geoff Brown
Bob Clary
Nicholas Nethercote
Csoregi Natalia
Cosmin Sabou
Edwin Takahashi
Andrew Halberstadt
Gabriele Svelto
Sylvestre Ledru
Ralph Giles
Henry Chang
Chris Manchester
Phil Ringnalda
Aidin Gharibnavaz
L. David Baron
Ed Morley