This file is also used by some browser-chrome tests which are still Python 2
for now. So let's not drop PY2 compat just yet.
Depends on D109728
Differential Revision: https://phabricator.services.mozilla.com/D111728
This file is also used by some browser-chrome tests which are still Python 2
for now. So let's not drop PY2 compat just yet.
Depends on D109728
Differential Revision: https://phabricator.services.mozilla.com/D111728
Under Python 3 on Windows we sometimes get an OSError which may be a
race with the process exiting. There's not much we can do about
exceptions during shutdown so just ignore them.
Differential Revision: https://phabricator.services.mozilla.com/D99434
Allow-list all Python code in tree for use with the black linter, and re-format all code in-tree accordingly.
To produce this patch I did all of the following:
1. Make changes to tools/lint/black.yml to remove include: stanza and update list of source extensions.
2. Run ./mach lint --linter black --fix
3. Make some ad-hoc manual updates to python/mozbuild/mozbuild/test/configure/test_configure.py -- it has some hard-coded line numbers that the reformat breaks.
4. Make some ad-hoc manual updates to `testing/marionette/client/setup.py`, `testing/marionette/harness/setup.py`, and `testing/firefox-ui/harness/setup.py`, which have hard-coded regexes that break after the reformat.
5. Add a set of exclusions to black.yml. These will be deleted in a follow-up bug (1672023).
# ignore-this-changeset
Differential Revision: https://phabricator.services.mozilla.com/D94045
Allow-list all Python code in tree for use with the black linter, and re-format all code in-tree accordingly.
To produce this patch I did all of the following:
1. Make changes to tools/lint/black.yml to remove include: stanza and update list of source extensions.
2. Run ./mach lint --linter black --fix
3. Make some ad-hoc manual updates to python/mozbuild/mozbuild/test/configure/test_configure.py -- it has some hard-coded line numbers that the reformat breaks.
4. Make some ad-hoc manual updates to `testing/marionette/client/setup.py`, `testing/marionette/harness/setup.py`, and `testing/firefox-ui/harness/setup.py`, which have hard-coded regexes that break after the reformat.
5. Add a set of exclusions to black.yml. These will be deleted in a follow-up bug (1672023).
# ignore-this-changeset
Differential Revision: https://phabricator.services.mozilla.com/D94045
Allow-list all Python code in tree for use with the black linter, and re-format all code in-tree accordingly.
To produce this patch I did all of the following:
1. Make changes to tools/lint/black.yml to remove include: stanza and update list of source extensions.
2. Run ./mach lint --linter black --fix
3. Make some ad-hoc manual updates to python/mozbuild/mozbuild/test/configure/test_configure.py -- it has some hard-coded line numbers that the reformat breaks.
4. Add a set of exclusions to black.yml. These will be deleted in a follow-up bug (1672023).
# ignore-this-changeset
Differential Revision: https://phabricator.services.mozilla.com/D94045
Use universal_newlines as a convenient way of treating the process I/O as text,
so string read/writes continue to succeed under python 3.
Differential Revision: https://phabricator.services.mozilla.com/D81774
Hopefully this will avoid some intermittent failures that occur when
`fix-stacks` is deleted at the end of a test run.
Differential Revision: https://phabricator.services.mozilla.com/D76980
Because the web platform test harness doesn't recognize them and annotates them
with `CRITICAL`, which is disconcerting, even though those errors are typically
not harmful.
Differential Revision: https://phabricator.services.mozilla.com/D71485
Currently AWSY-with-DMD doesn't work on Windows. This is because `fix-stacks`
is initialized lazily, and by the time the initialization happens some file
descriptors for files are open, and that leads to some major Python2-on-Windows
sadness as described in the big comment in the commit.
To fix the problem, this commit adds an `init` function to `fix_stacks.py` so
that `fix-stacks` can be initialized eagerly, hopefully before any file
descriptors for files are open.
For `dmd.py`, other than fixing the AWSY problems, this has little effect,
because `fix-stacks` is always initialized.
For `utils.py`, which is used to process the output of most tests, this has a
more noticeable effect: the `fix-stacks` process is always spawned, rather than
being spawned only when needed. If no stack traces appear in the test output,
this means that `fix-stacks` is spawned unnecessarily. But it's cheap to spawn;
the expensive part only happens when stack traces start getting fixed. So I
think this change in behaviour is acceptable.
Furthermore, the commit adds a `finish` function to `fix_stacks.py`, so that
the `fix-stacks` process can be explicitly shut down. This has never been done
for processes spawned for any of the stack fixing scripts. It's never caused
problems on Linux/Mac, but it seems to be necessary on Windows to avoid
similar "this file is locked" problems with the test_dmd.js test.
The commit also renames some things to more standard Python style, e.g.
`json_mode` instead of `jsonMode`.
Finally, Android tests use `utils.py` from the repository but `fix_stacks.py`
from the Android host utils. Because the two scripts must be updated in tandem,
this commit also updates the Android host utils to a version that contains the
updated `fix_stacks.py`. Thanks to aerickson for packaging up the new Android
host utils and providing the change to the `hostutils.manifest` file.
Differential Revision: https://phabricator.services.mozilla.com/D69478
This commit removes `test_fix_stack_using_bpsyms.py`. That test can't easily be
modified to work with `fix_stacks.py` because it relies on internal
implementation details of `fix_stack_using_bpsym.py`. The unit testing done in
the `fix-stacks` repo provides test coverage that is as good or better.
Differential Revision: https://phabricator.services.mozilla.com/D66924
--HG--
extra : moz-landing-system : lando
Currently AWSY-with-DMD doesn't work on Windows. This is because `fix-stacks`
is initialized lazily, and by the time the initialization happens some file
descriptors for files are open, and that leads to some major Python2-on-Windows
sadness as described in the big comment in the commit.
To fix the problem, this commit adds an `init` function to `fix_stacks.py` so
that `fix-stacks` can be initialized eagerly, hopefully before any file
descriptors for files are open.
For `dmd.py`, other than fixing the AWSY problems, this has little effect,
because `fix-stacks` is always initialized.
For `utils.py`, which is used to process the output of most tests, this has a
more noticeable effect: the `fix-stacks` process is always spawned, rather than
being spawned only when needed. If no stack traces appear in the test output,
this means that `fix-stacks` is spawned unnecessarily. But it's cheap to spawn;
the expensive part only happens when stack traces start getting fixed. So I
think this change in behaviour is acceptable.
The commit also renames some things to more standard Python style, e.g.
`json_mode` instead of `jsonMode`.
Differential Revision: https://phabricator.services.mozilla.com/D69478
--HG--
extra : moz-landing-system : lando
Now that `fix_stacks.py` is being used instead of `fix_macosx_stack.py`,
stack-fixing time has dropped from about 14 minutes to about 30 seconds on my
new MacBook Pro.
Also, print a warning about stacks not being fixed if `MOZ_DISABLE_STACK_FIX`
is set. This warning shows up at the start of the test run.
Also, print a warning about stack fixing slowness, because 30 seconds is long
enough to possibly be surprising. This warning shows up just before the first
stack frame is fixed, like this:
```
Assertion failure: false (BEEP BOOP), at /home/njn/moz/au3/dom/base/nsGlobalWindowOuter.cpp:1342
Initializing stack-fixing for the first stack frame, this may take a while...
#01: nsGlobalWindowOuter::nsGlobalWindowOuter(unsigned long) (/home/njn/moz/au3/dom/base/nsGlobalWindowOuter.cpp:1342)
#02: ...
```
Differential Revision: https://phabricator.services.mozilla.com/D65676
--HG--
extra : moz-landing-system : lando
Specifically:
- handle input that doesn't end in a newline;
- only pass lines that match the regexp to `fix-stacks`.
These changes are needed for `utils.py` to use `fix_stacks.py` without
problems.
Differential Revision: https://phabricator.services.mozilla.com/D65944
--HG--
extra : moz-landing-system : lando
On Linux and Mac, this makes `dmd.py` *much* faster when it is first run on a
DMD data file.
On Windows, this makes DMD actually usable locally. Previously the stacks
weren't fixed and so were rubbish.
Differential Revision: https://phabricator.services.mozilla.com/D57271
--HG--
extra : moz-landing-system : lando
So that dmd.py isn't broken by function names that, after stack fixing, contain
escape-worthy chars such as " and /.
--HG--
extra : rebase_source : 625cef56e4599bbb810273ef04af2b971e777b58
When reworking the script each entry holding a function name was replaced by a
dictionary holding both the function name and its size. This significantly
increased memory consumption as using a full-fledged dictionary for only two
fields is very space inefficient. This patch uses a named tuple instead of a
dictionary for every entry, reducing memory consumption by almost four times.
Differential Revision: https://phabricator.services.mozilla.com/D6603
--HG--
extra : moz-landing-system : lando
This patch changes the way we search symbols when fixing up a stack.
Previously we would find the closest PUBLIC or FUNC entry lower than a given
address. Because of how symbol files were processed we preferred PUBLIC
entries to FUNC ones. Now we look first for the function that contains the
address (obtained from the FUNC entries) then if none is available we look for
the closest, lower PUBLIC entry.
Differential Revision: https://phabricator.services.mozilla.com/D5033
--HG--
extra : moz-landing-system : lando
This includes both the vanilla sources we haven't forked and the client
sources that we have. Client patches were applied manually up to version
69c2c51dd89965d234eec16e3a9353634831916b. The following changes were not
included as they break merging segments corresponding to libxul.so in the
module list:
8915f7be39448d9257b6da3ad0233944d1d9a92a
17ad0c18b179c135fc5a3d2bba199c3fa4276035
94b6309aecaddfcf11672f6cfad9575d68ad3b40
With these changes applied two entries for libxul.so are generated, the second
one is bogus and prevents symbolication from working correctly.
The build system and some of the tools relying on breakpad were also updated
to work with the new version.
--HG--
extra : source : fe4d49307f8890a0c430c257c96f74a9552eeb31
extra : histedit_source : bc84861445bd93856cd0d0c864fd15ad7d9ccc12%2C1efd65797da46e33481afa61a302098780b0f107
These locations have changed since the end of Google Code.
MozReview-Commit-ID: FSGhFBDaTCq
--HG--
extra : rebase_source : 4ef004b2d0b8f1a6331db0f44ccf70d05ad62774
If the symbols file doesn't give a function name, substitute
'unnamed_function'. A more complete fix would be to catch
all ValueError exceptions and continue sensibly, but this
addresses the immediate issue with the output from `rustc -g`.
MozReview-Commit-ID: 666ruvLlJ5t
This patch introduces a small utility program to extract a guid from a shared library
or executable on windows to identify the correct symbol file to read in fix_stack_using_bpsyms.py.
In order for this to work correctly on windows, the library name provided by
MozDescribeCodeAddress needs to be a full path, so the LoadedImageName field
from the IMAGEHLP_MODULE64 structure is used here instead of the ModuleName
field.
MozReview-Commit-ID: 8zkfLWjKVs2
Mike Hommey
Andrew Halberstadt
Sandor Molnar
Ben Hearsum
Bogdan Tara
Butkovits Atila
Brindusan Cristian
Emilio Cobos Álvarez
Paul Bone
James Graham
Ricky Stewart
Dorel Luca
Geoff Brown
Bob Clary
Nicholas Nethercote
Csoregi Natalia
Cosmin Sabou
Edwin Takahashi
Andrew Halberstadt
Gabriele Svelto
Sylvestre Ledru
Ralph Giles
Henry Chang
Chris Manchester
Phil Ringnalda