Currently AWSY-with-DMD doesn't work on Windows. This is because `fix-stacks`
is initialized lazily, and by the time the initialization happens some file
descriptors for files are open, and that leads to some major Python2-on-Windows
sadness as described in the big comment in the commit.
To fix the problem, this commit adds an `init` function to `fix_stacks.py` so
that `fix-stacks` can be initialized eagerly, hopefully before any file
descriptors for files are open.
For `dmd.py`, other than fixing the AWSY problems, this has little effect,
because `fix-stacks` is always initialized.
For `utils.py`, which is used to process the output of most tests, this has a
more noticeable effect: the `fix-stacks` process is always spawned, rather than
being spawned only when needed. If no stack traces appear in the test output,
this means that `fix-stacks` is spawned unnecessarily. But it's cheap to spawn;
the expensive part only happens when stack traces start getting fixed. So I
think this change in behaviour is acceptable.
Furthermore, the commit adds a `finish` function to `fix_stacks.py`, so that
the `fix-stacks` process can be explicitly shut down. This has never been done
for processes spawned for any of the stack fixing scripts. It's never caused
problems on Linux/Mac, but it seems to be necessary on Windows to avoid
similar "this file is locked" problems with the test_dmd.js test.
The commit also renames some things to more standard Python style, e.g.
`json_mode` instead of `jsonMode`.
Finally, Android tests use `utils.py` from the repository but `fix_stacks.py`
from the Android host utils. Because the two scripts must be updated in tandem,
this commit also updates the Android host utils to a version that contains the
updated `fix_stacks.py`. Thanks to aerickson for packaging up the new Android
host utils and providing the change to the `hostutils.manifest` file.
Differential Revision: https://phabricator.services.mozilla.com/D69478
This commit removes `test_fix_stack_using_bpsyms.py`. That test can't easily be
modified to work with `fix_stacks.py` because it relies on internal
implementation details of `fix_stack_using_bpsym.py`. The unit testing done in
the `fix-stacks` repo provides test coverage that is as good or better.
Differential Revision: https://phabricator.services.mozilla.com/D66924
--HG--
extra : moz-landing-system : lando
Currently AWSY-with-DMD doesn't work on Windows. This is because `fix-stacks`
is initialized lazily, and by the time the initialization happens some file
descriptors for files are open, and that leads to some major Python2-on-Windows
sadness as described in the big comment in the commit.
To fix the problem, this commit adds an `init` function to `fix_stacks.py` so
that `fix-stacks` can be initialized eagerly, hopefully before any file
descriptors for files are open.
For `dmd.py`, other than fixing the AWSY problems, this has little effect,
because `fix-stacks` is always initialized.
For `utils.py`, which is used to process the output of most tests, this has a
more noticeable effect: the `fix-stacks` process is always spawned, rather than
being spawned only when needed. If no stack traces appear in the test output,
this means that `fix-stacks` is spawned unnecessarily. But it's cheap to spawn;
the expensive part only happens when stack traces start getting fixed. So I
think this change in behaviour is acceptable.
The commit also renames some things to more standard Python style, e.g.
`json_mode` instead of `jsonMode`.
Differential Revision: https://phabricator.services.mozilla.com/D69478
--HG--
extra : moz-landing-system : lando
Each allocation page is now bracketed by a guard page, and allocations are put
at the end of their page so that bounds violations trigger a crash.
Various operations (realloc(), free(), malloc_usable_size()) now require that
the pointer they are given points to the start of an allocation.
Differential Revision: https://phabricator.services.mozilla.com/D43842
--HG--
rename : toolkit/crashreporter/test/unit_ipc/test_content_phc2.js => toolkit/crashreporter/test/unit_ipc/test_content_phc3.js
extra : moz-landing-system : lando
This is in preparation for the introduction of "guard pages", which are
interleaved with alloc pages. The specific renamings are:
- kMaxPageAllocs --> kNumAllocPages
- PagePtr --> AllocPagePtr
- PageState --> AllocPageState
- PageInfo --> AllocPageInfo
- mPages --> mAllocPages
- AssertPageInUse --> AssertAllocPageInUse
Differential Revision: https://phabricator.services.mozilla.com/D43841
--HG--
extra : moz-landing-system : lando
If we don't find the block, just print out the original argument,
instead of the version converted to an integer. Python gets mad when
you try to concatenate an int and a string.
Differential Revision: https://phabricator.services.mozilla.com/D67404
--HG--
extra : moz-landing-system : lando
Apps targeting SDK 29 are not allowed to open /dev/ashmem directly, and
instead must use NDK functions. Those functions are only available in
SDK 26 and higher, so we need this shim to use the functions if they
are available, else fallback to opening /dev/ashmem directly.
Differential Revision: https://phabricator.services.mozilla.com/D61012
--HG--
extra : moz-landing-system : lando
On Linux and Mac, this makes `dmd.py` *much* faster when it is first run on a
DMD data file.
On Windows, this makes DMD actually usable locally. Previously the stacks
weren't fixed and so were rubbish.
Differential Revision: https://phabricator.services.mozilla.com/D57271
--HG--
extra : moz-landing-system : lando
Apps targeting SDK 29 are not allowed to open /dev/ashmem directly, and
instead must use NDK functions. Those functions are only available in
SDK 26 and higher, so we need this shim to use the functions if they
are available, else fallback to opening /dev/ashmem directly.
Differential Revision: https://phabricator.services.mozilla.com/D61012
--HG--
extra : moz-landing-system : lando
With the old constants the page alloc slots would fill up quickly and the hit
rate would quickly drop below 20%.
With the new constants the alloc slots don't fill up so quickly and the hit
rate remains at or near 100% for a lot longer. Also, page allocs are recycled
more slowly, which should increase the likelihood of UAFs being detected
correctly.
Differential Revision: https://phabricator.services.mozilla.com/D62539
--HG--
extra : moz-landing-system : lando
Specifically, the number of page allocs in use, and the page alloc hit rate.
Differential Revision: https://phabricator.services.mozilla.com/D62538
--HG--
extra : moz-landing-system : lando
The assert that was added in bug 1610720 assumed the node links were
reset when a node is removed from a RedBlackTree, but that wasn't the
case. We can either remove the assert, or clear node links. We pick the
latter.
Differential Revision: https://phabricator.services.mozilla.com/D61515
--HG--
extra : moz-landing-system : lando
Currently, it's expected to "lazily" dispose of the arena. That is, if
the arena still has allocations, the arena is kept around. Using it for
new allocations is an error, but otherwise, it's possible to free
allocations from there. Once it's empty, the arena is freed too. Well,
that's the idea, but that last part is not actually implemented, cf.
bug 1364359.
While that is still an (optionally) desirable disposal mode, other modes
would be useful as well. So this bug is here to switch the current mode
to something that can more easily be used for e.g. bug 1377999.
The idea is to make moz_dispose_arena() free the arena immediately, and
MOZ_CRASH if it's not already empty (on a besst effort basis).
Differential Revision: https://phabricator.services.mozilla.com/D60635
--HG--
extra : moz-landing-system : lando
The base_node_alloc and base_node_dealloc are wrappers for the base
allocator that allow to free memory (the base allocator doesn't support
freeing memory), using a free list. They are dedicated to extent_node_t
objects (the only ones we currently ever need to free).
We're going to need to free other types of objects, so make this more
generic.
Differential Revision: https://phabricator.services.mozilla.com/D60634
--HG--
extra : moz-landing-system : lando
This was done by:
This was done by applying:
```
diff --git a/python/mozbuild/mozbuild/code-analysis/mach_commands.py b/python/mozbuild/mozbuild/code-analysis/mach_commands.py
index 789affde7bbf..fe33c4c7d4d1 100644
--- a/python/mozbuild/mozbuild/code-analysis/mach_commands.py
+++ b/python/mozbuild/mozbuild/code-analysis/mach_commands.py
@@ -2007,7 +2007,7 @@ class StaticAnalysis(MachCommandBase):
from subprocess import Popen, PIPE, check_output, CalledProcessError
diff_process = Popen(self._get_clang_format_diff_command(commit), stdout=PIPE)
- args = [sys.executable, clang_format_diff, "-p1", "-binary=%s" % clang_format]
+ args = [sys.executable, clang_format_diff, "-p1", "-binary=%s" % clang_format, '-sort-includes']
if not output_file:
args.append("-i")
```
Then running `./mach clang-format -c <commit-hash>`
Then undoing that patch.
Then running check_spidermonkey_style.py --fixup
Then running `./mach clang-format`
I had to fix four things:
* I needed to move <utility> back down in GuardObjects.h because I was hitting
obscure problems with our system include wrappers like this:
0:03.94 /usr/include/stdlib.h:550:14: error: exception specification in declaration does not match previous declaration
0:03.94 extern void *realloc (void *__ptr, size_t __size)
0:03.94 ^
0:03.94 /home/emilio/src/moz/gecko-2/obj-debug/dist/include/malloc_decls.h:53:1: note: previous declaration is here
0:03.94 MALLOC_DECL(realloc, void*, void*, size_t)
0:03.94 ^
0:03.94 /home/emilio/src/moz/gecko-2/obj-debug/dist/include/mozilla/mozalloc.h:22:32: note: expanded from macro 'MALLOC_DECL'
0:03.94 MOZ_MEMORY_API return_type name##_impl(__VA_ARGS__);
0:03.94 ^
0:03.94 <scratch space>:178:1: note: expanded from here
0:03.94 realloc_impl
0:03.94 ^
0:03.94 /home/emilio/src/moz/gecko-2/obj-debug/dist/include/mozmemory_wrap.h:142:41: note: expanded from macro 'realloc_impl'
0:03.94 #define realloc_impl mozmem_malloc_impl(realloc)
Which I really didn't feel like digging into.
* I had to restore the order of TrustOverrideUtils.h and related files in nss
because the .inc files depend on TrustOverrideUtils.h being included earlier.
* I had to add a missing include to RollingNumber.h
* Also had to partially restore include order in JsepSessionImpl.cpp to avoid
some -WError issues due to some static inline functions being defined in a
header but not used in the rest of the compilation unit.
Differential Revision: https://phabricator.services.mozilla.com/D60327
--HG--
extra : moz-landing-system : lando
rg -l 'mozilla/Move.h' | xargs sed -i 's/#include "mozilla\/Move.h"/#include <utility>/g'
Further manual fixups and cleanups to the include order incoming.
Differential Revision: https://phabricator.services.mozilla.com/D60323
--HG--
extra : moz-landing-system : lando
This is required on some systems such as Fedora to allow
building with -O0 together with --warnings-as-errors due to
a check in /usr/include/features.h
Differential Revision: https://phabricator.services.mozilla.com/D56696
--HG--
extra : moz-landing-system : lando
Currently the PHC code uses char* and uintptr_t in various address
computations. This patch changes it to use uint8_t* instead, which is clearer
than char* and avoids the need for various casts.
Differential Revision: https://phabricator.services.mozilla.com/D43840
--HG--
extra : moz-landing-system : lando
glibc marks various allocation functions as `throw()`. This addition
hasn't been a problem until we tried to enable C++17, where clang
started complaining that we were redeclaring functions with mismatched
exception specifications. Peculiarly, glibc declares virtually
everything we redeclare as `throw()`, but clang only complains about the
mismatches for a particular subset of functions.
The approach taken in this patch is to add another potentially defined
macro to malloc_decls.h, `NOTHROW_MALLOC_DECL`. This macro works
exactly like `MALLOC_DECL`, except that clients can define
`NOTHROW_MALLOC_DECL` to add appropriate `throw()` specifiers when
declaring functions at global scope, and thereby avoid mismatched
exception specifications.
Differential Revision: https://phabricator.services.mozilla.com/D44045
--HG--
extra : moz-landing-system : lando
In the MinGW build, calls to malloc inside mozglue were not being
redirected as defined in the .def file. We create aliases for the
redirected functions to correctly redirect them inside mozglue.
An alternate solution for this exists. Rather than creating the
importlib during the linking step for mozglue, we could have used
dlltool to create it, and then provided it during linking. This
would allow mozglue to know that it should redirect calls to malloc
to je_malloc as specified in the .def file.
Differential Revision: https://phabricator.services.mozilla.com/D38407
--HG--
extra : moz-landing-system : lando
Sylvestre Ledru
Cosmin Sabou
Nicholas Nethercote
pbz
Csoregi Natalia
Bogdan Tara
André Bargull
Simon Giesecke
Andrew McCreight
Mark Banner
James Willcox
Ciure Andrei
Masatoshi Kimura
Coroiu Cristina
Mike Hommey
Emilio Cobos Álvarez
Brian Hackett
Tom Ritter
Andreea Pavel
Gian-Carlo Pascutto
Christian Holler
Nathan Froyd