Allow-list all Python code in tree for use with the black linter, and re-format all code in-tree accordingly.
To produce this patch I did all of the following:
1. Make changes to tools/lint/black.yml to remove include: stanza and update list of source extensions.
2. Run ./mach lint --linter black --fix
3. Make some ad-hoc manual updates to python/mozbuild/mozbuild/test/configure/test_configure.py -- it has some hard-coded line numbers that the reformat breaks.
4. Add a set of exclusions to black.yml. These will be deleted in a follow-up bug (1672023).
# ignore-this-changeset
Differential Revision: https://phabricator.services.mozilla.com/D94045
Today we don't require that `mach` `CommandProvider`s subclass from any particular parent class and we're very lax about the requirements they must meet. While that's convenient in certain circumstances, it has some unfortunate implications for feature development.
Today the only requirements that we have for `CommandProvider`s are that they have an `__init__()` method that takes either 1 or 2 arguments, the second of which must be called `context` and is populated with the `mach` `CommandContext`. Again, while this flexibility is occasionally convenient, it is limiting. As we add features to `mach`, having a better idea what the shape of our `CommandProvider`s are and how we can instantiate them and use them is increasingly important, and this gives us additional control when having `mach` configure `CommandProvider`s based on data that is only available at the `mach` level. In particular, we plan to leverage this in bugs 985141 and 1654074.
Here we add validation to the `CommandProvider` decorator to ensure all classes inherit from `MachCommandBase`, update all `CommandProvider`s in-tree to inherit from `MachCommandBase`, and update source and test code accordingly.
Follow-up work: we now require (de facto) that the `context` be populated with a `topdir` attribute by the `populate_context_handler` function, since instantiating the `MachCommandBase` requires a `topdir` be provided. This is fine for now in the interest of keeping this patch reasonably sized, but some additional refactoring could make this cleaner.
Differential Revision: https://phabricator.services.mozilla.com/D86255
This has the side effect of not initializing fontconfig before the
valgrind test itself runs, which changes the code path leading to
`FcConfigAddDirList`, which eventually leads to suppressed leaks.
Those leaks are then not discarded because the caller doesn't match what
is in the suppression file anymore, so we remove the caller.
Differential Revision: https://phabricator.services.mozilla.com/D85353
And add suppressions for the new errors that this adds to the valgrind
run. It's not clear if it's a legitimate thing that LLVM does that
valgrind doesn't cope with, like many others, but the fact is running
valgrind on a PGO/LTO build doesn't show these errors, so we're not
actually shipping them anyways (but does show _different_ errors that we
don't see on the build the valgrind jobs do)
Differential Revision: https://phabricator.services.mozilla.com/D81016
Create a new MachCommandCondition, "is_firefox_or_thunderbird" which is then used
to allow mach valgrind-test work for Thunderbird builds.
Differential Revision: https://phabricator.services.mozilla.com/D73153
Otherwise, Valgrind is liable to see false positives from mismatched
`new` where the `delete` has been inlined to `free` or vice versa.
Differential Revision: https://phabricator.services.mozilla.com/D55553
--HG--
extra : moz-landing-system : lando
Valgrind report:
```
TEST-UNEXPECTED-FAIL | valgrind-test | 192 bytes in 3 blocks are definitely lost at malloc / dl_open_worker / _dl_catch_error / _dl_open
==2561== 192 bytes in 3 blocks are definitely lost in loss record 348 of 399
==2561== at 0x4C2B240: malloc+112 (vg_replace_malloc.c:298)
==2561== by 0x4012919: dl_open_worker+1977 (dl-open.c:457)
==2561== by 0x400DD55: _dl_catch_error+101 (dl-error.c:178)
==2561== by 0x4011CC9: _dl_open+185 (dl-open.c:633)
==2561== by 0x5051F65: dlopen_doit+101 (dlopen.c:67)
==2561== by 0x400DD55: _dl_catch_error+101 (dl-error.c:178)
==2561== by 0x50522EB: _dlerror_run+123 (dlerror.c:164)
==2561== by 0x5051EE0: dlopen@@GLIBC_2.2.5+48 (dlopen.c:88)
==2561== by 0x1148FC: GetLibHandle (xpcom/glue/standalone/nsXPCOMGlue.cpp:86)
==2561== by 0x1148FC: ReadDependentCB (xpcom/glue/standalone/nsXPCOMGlue.cpp:136)
==2561== by 0x1148FC: XPCOMGlueLoad (xpcom/glue/standalone/nsXPCOMGlue.cpp:306)
==2561== by 0x1148FC: mozilla::GetBootstrap(char const*, mozilla::LibLoadingStrategy)+572 (xpcom/glue/standalone/nsXPCOMGlue.cpp:374)
==2561== by 0x114213: InitXPCOMGlue(mozilla::LibLoadingStrategy)+131 (browser/app/nsBrowserApp.cpp:223)
==2561== by 0x113E6B: main+219 (browser/app/nsBrowserApp.cpp:284)
```
Nothing due directly to this patch, so it is likely a dlopen issue.
Differential Revision: https://phabricator.services.mozilla.com/D33904
--HG--
extra : moz-landing-system : lando
The exit code from valgrind may subtly indicate how it failed (like, if
it's 139 or 137, which would mean respectively segfault or killed by
something external), which is currently completely hidden, making
diagnostics of things like bug 1545094 harder.
Differential Revision: https://phabricator.services.mozilla.com/D32412
--HG--
extra : moz-landing-system : lando
When valgrind prints out backtraces, it prints raw addresses and symbol
names, but that doesn't help find the exact code that caused the errors,
because we don't know where the libraries are loaded.
With --sym-offsets=yes, it adds the offset from the symbol, which allows
to find the relevant code in the binary.
This serves two purposes:
1) It makes web-platform-tests pref downloading/handling a little more robust. When
run externally, it now downloads the entire testing/profiles directory. When loading
prefs it will look for both prefs_general.js (to support older versions of Firefox)
and profiles.json (for support moving forward).
This way we can add/remove/rename pref files under these directories without needing
to worry about breaking upstream wpt.
2) It provides developers an overview of which harnesses are using which base profiles.
Instead of hunting through test harness code to find this information, they can glance
at profiles.json.
MozReview-Commit-ID: AMzdnD8aGA2
--HG--
extra : rebase_source : 6fa0a802680417e49fcef99f3d03de7458a8fcba
This moves testing/profiles/prefs_general.js to
testing/profiles/common/user.js. It also adds an 'extensions' folder to the
common profile. Dropping extension files here will get them installed in all
test harnesses (useful for testing on try).
The idea is that all test harnesses will eventually use this 'common' profile.
We'll also create some new per harness profiles, e.g testing/profiles/mochitest
and testing/profiles/reftest. This way there will be a single location
developers can go to set preferences, both for a specific harness, and across
all harnesses.
MozReview-Commit-ID: 8sqBqLiypgU
--HG--
rename : testing/profiles/prefs_general.js => testing/profiles/common/user.js
extra : rebase_source : 72a4a4b691e93c77479c7e70647b0ca373862c51
This moves testing/profiles/prefs_general.js to
testing/profiles/common/user.js. It also adds an 'extensions' folder to the
common profile. Dropping extension files here will get them installed in all
test harnesses (useful for testing on try).
The idea is that all test harnesses will eventually use this 'common' profile.
We'll also create some new per harness profiles, e.g testing/profiles/mochitest
and testing/profiles/reftest. This way there will be a single location
developers can go to set preferences, both for a specific harness, and across
all harnesses.
MozReview-Commit-ID: 8sqBqLiypgU
--HG--
rename : testing/profiles/prefs_general.js => testing/profiles/common/user.js
extra : rebase_source : 7599913e547533f2f57b597ad0f238c6cd391c82
The existing suppression tries to be specific about the callstack that
arrives at the function, but that breaks when we rejigger the machinery
up the callstack. In practice, the existing suppressions cover most of
the ways we would arrive at selector parsing, and so I think the
specificity here is more trouble than it's worth.
MozReview-Commit-ID: 2k52xq64SQX
This patch adds suppressions as needed on automation, for Servo-in-Gecko as
compiled by rustc 1.25. It appears there are only three false error sites
fun:*style*values*specified*color*Color*style*parser*Parse*parse*
fun:*selectors*parser*SelectorList*Impl*parse*
fun:*style*properties*shorthands*
but there are a number of different paths leading to them, especially the
first, hence the use of 17 suppressions in total, so as to remain specific.
Following some investigation of the machine code involved, I think these are
all Valgrind/Memcheck false positives, unfortunately, and probably to do
with (legitimate) operand swapping in && or || expressions by rustc+LLVM.
MozReview-Commit-ID: EpDmb4PEyoy
In nsFrame::ComputeSize and nsFrame::ComputeSizeWithIntrinsicDimensions, the
following expressions
isFlexItem && usingFlexBasisForISize
isFlexItem && !usingFlexBasisForISize
are sometimes compiled by recent gcc/clang in the opposite order, viz
[!]usingFlexBasisForISize && isFlexItem. In this case the transformation is
correct as can be seen by analysing code earlier in these functions.
Unfortunately this causes Valgrind/Memcheck to report a branch on undefined
data which, in this case, is a false positive.
A simple fix is simply to initialise usingFlexBasisForISize to false at its
declaration point.
--HG--
extra : rebase_source : 39877e4ea8ec678288e6b49af120445c96ef8c0a
The first two are variants of existing ones, with a slightly different
signature, and tha last one is well known and found in glib-related
suppression files like https://github.com/flatpak/flatpak/blob/master/tests/glib.supp
--HG--
extra : rebase_source : 2a209cc5987405e7ff638bbf69e1b74202b0357e
The MACHTYPE bash variable is an odd thing that returns e.g.
x86_64-redhat-linux-gnu on a CentOS system, but x86_64-pc-linux-gnu
on a Debian system, and possibly something different on other distros.
mach valgrind-test is the only place actually relying on MACHTYPE.
Others rely on information from python modules. Uniformize that, and use
the more generic 'pc' rather than 'redhat'.
--HG--
rename : build/valgrind/i386-redhat-linux-gnu.sup => build/valgrind/i386-pc-linux-gnu.sup
rename : build/valgrind/x86_64-redhat-linux-gnu.sup => build/valgrind/x86_64-pc-linux-gnu.sup
extra : rebase_source : ad94ce69e8094d2b9ddae97a3d261945886c0a61
We believe this is another spurious memcheck error triggered
by code from Rust 1.23.0. See also bug 1394696.
For some reason, this error occurs both with and without
the leading underscore on mangled std::sync::once methods,
so this change matches either with a wildcard.
MozReview-Commit-ID: 4upSAPqAtNA
--HG--
extra : rebase_source : 5f697aaa5e170369f08d385d10c1aac9d8c1e50b
LLVM can optimize code to a form `if B && A` when A is
always false but B is undefined. This triggers a valgrind
memcheck warning since the conditional depends on undefined
data but in practice it can never have side-effects.
Rust 1.20.0 seems to trigger this in the Option code. Since
we believe the transform is correct in these cases and
valgrind is incorrect to warn, we surpress the error.
Thanks to Julian Seward for the analysis and help
writing the suppression entries.
MozReview-Commit-ID: pF1Bmy5PRY
This removes the double-include macro hackery that we use to define two
separate string types (nsAString and nsACString) in favor of a templated
solution.
Annotations for Valgrind and the JS hazard analysis are updated as well as
the rust binding generations for string code.
--HG--
extra : rebase_source : 63ab2c4620cfcd4b764d42d654c82f30f984d016
extra : source : 9115364cd4aa078c49bba7911069f8178e55166f
Using /home/worker is the build directory has a 30% talos performance
loss, because test machines has a /home mount directory.
MozReview-Commit-ID: 554IPMRWgzK
--HG--
extra : rebase_source : 00827d3f6bd705419bc801eb05b543af1ddc274f
This patch:
* Adds a suppression for some leaks in libLLVM-3.6-mesa.so.
* Adds Valgrind flag --keep-debuginfo=yes so that the abovementioned leak
stacks can be symbolised and hence suppressed even after
libLLVM-3.6-mesa.so is unmapped from the process.
* Adds Valgrind flag --expensive-definedness-checks=yes as an attempt to
reduce Memcheck false positives from LLVM and rustc compiled code. This
change is aimed primarily at bug 1365915.
MozReview-Commit-ID: KiOZG2O8wzs
Bug 1338651 was backed out because when building a newer image, there
was a valgrind leak report that couldn't resolve symbols. Further
investigation showed the valgrind package installed had symbols stripped.
We upgrade valgrind version and build it from source with symbols.
We had to build inside the docker image because we need to run
"make install". Using "make dist" to generate a tar ball will also run
"make docs", and it is hard to make it work because of the outdated
texlive package present in CentOS 6.
We also apply a patch [1] to valgrind correctly generate symbols
for unloaded objects.
[1] https://bugs.kde.org/show_bug.cgi?id=79362#c62
MozReview-Commit-ID: 2IhuJY28Ke3
It was used for valgrind buildbot jobs, but those have been gone since
bug 1278611, and the taskcluster jobs use mozharness.
--HG--
extra : rebase_source : b9d39b1515a677d0662c2f13f21d83b953e70054
This additionally changes exit() calls with |return VALUE| so
that we are sure to call the destructors and valgrind doesn't complain.
Moreover, this disables the 'new-profile' ping when Firefox is ran
on valgrind.
MozReview-Commit-ID: BlGE9w6yGCL
--HG--
extra : rebase_source : d21b7404ac8dba6f3664f0f8d375429a0dec0ee4
Valgrind-test's output handler buffers valgrind's output to detect
errors and emit proper messages that are detected by automation log
parsing. When an error is detected, the buffered output is reinjected,
but there was a typo that prevented that from actually happening,
which ate the first lines of stack traces.
This will add more verbosity, with the intent of having the triggered
suppressions be displayed in the form:
"--23983-- used_suppression: 3 Bug 794372 /builds/slave/try-l64-valgrind-0000000000000/src/build/valgrind/cross-architecture.sup:90 suppressed: 20,736 bytes in 648 blocks"
This removes ambiguity as to which modules are being imported, making
import slightly faster as Python doesn't need to test so many
directories for file presence.
All files should already be using absolute imports because mach command
modules aren't imported to the package they belong to: they instead
belong to the "mach" package. So relative imports shouldn't have been
used.
--HG--
extra : commitid : 6tFME1KKfTD
extra : rebase_source : 78728f82f5487281620e00c2a8004cd5e1968087
Ricky Stewart
Mike Hommey
Jonathan Kew
Rob Lemley
Tooru Fujisawa
Chris Manchester
Nathan Froyd
Emilio Cobos Álvarez
Dustin J. Mitchell
Razvan Maries
Gerald Squelart
Gijs Kruitbosch
Andreea Pavel
Julian Seward
Sylvestre Ledru
Andrew Halberstadt
Aaron Klotz
Bogdan Tara
Bobby Holley
Xidorn Quan
Ryan VanderMeulen
Ralph Giles
Eric Rahm
Wander Lairson Costa
J. Ryan Stinnett
Sebastian Hengst
Alessio Placitelli
Jeff Muizelaar
David Major
Martin Stransky
Matt Woodrow
Carsten "Tomcat" Book
Karl Tomlinson
Nicholas Nethercote
Julien Pagès
Dão Gottwald
Jean-Yves Avenard
Georg Fritzsche
Gregory Szorc