Introduces "./mach remote vendor-puppeteer" for vendoring the
Puppeteer client without dependencies into remote/test/puppeteer/.
The particular checkout of Puppeteer is
https://github.com/andreastt/puppeteer/tree/firefox, which contains a
couple of hotfixes we need for the client to work with the Firefox
implementation of CDP.
The remote agent targets a specific version of Puppeteer, so it is
not suitable for this to be vendored under third_party/. We also
wouldn't want other code in central to accidentally use a patched fork.
The vendoring process is not part of "./mach vendor" because it does
not yet have Node.js support, and implementing that for mach is outside
the scope of getting the Puppeteer tests running with the remote agent.
Differential Revision: https://phabricator.services.mozilla.com/D37007
--HG--
extra : moz-landing-system : lando
The remaining uses all need adjustements to in-tree mozconfigs, so they
all need to be done at once.
However, to make things slightly more intelligible, we do this in two
steps. This is step 1: we modify the use_toolchain transform to take care of
the transformation, while keeping the task definitions intact, so that
we only deal with mozconfig and build script adjustements here.
Differential Revision: https://phabricator.services.mozilla.com/D41890
Introduces "./mach remote vendor-puppeteer" for vendoring the
Puppeteer client without dependencies into remote/test/puppeteer/.
The particular checkout of Puppeteer is
https://github.com/andreastt/puppeteer/tree/firefox, which contains a
couple of hotfixes we need for the client to work with the Firefox
implementation of CDP.
The remote agent targets a specific version of Puppeteer, so it is
not suitable for this to be vendored under third_party/. We also
wouldn't want other code in central to accidentally use a patched fork.
The vendoring process is not part of "./mach vendor" because it does
not yet have Node.js support, and implementing that for mach is outside
the scope of getting the Puppeteer tests running with the remote agent.
Differential Revision: https://phabricator.services.mozilla.com/D37007
--HG--
extra : moz-landing-system : lando
Just like C++14 sized deallocation support, we don't want to support
this. We shouldn't be using `new` on over-aligned types anyway.
Differential Revision: https://phabricator.services.mozilla.com/D41819
--HG--
extra : moz-landing-system : lando
As shown by python/mozbuild/mozbuild/test/test_preprocessor.py, whitespace is
allowed within expressions, and chained #if/#elif/#else sequences work as you'd
expect.
Differential Revision: https://phabricator.services.mozilla.com/D41494
--HG--
extra : moz-landing-system : lando
There are about the same number of occurrences of "ENABLE_INTL_API" and "EXPOSE_INTL_API"
in the tree, so preferring one over the other doesn't lead to fewer changes. Therefore
I went with "ENABLE_INTL_API", because "ENABLE_" (resp. "MOZ_ENABLE") is already used as
the prefix for other preprocessor ifdef's.
Differential Revision: https://phabricator.services.mozilla.com/D41188
--HG--
extra : moz-landing-system : lando
All .xul files have been loading as HTMLDocuments for a few weeks now, so
it should be safe to remove the XULDocument implementation.
Differential Revision: https://phabricator.services.mozilla.com/D41238
--HG--
extra : moz-landing-system : lando
Some recent changes to how we set cross-language LTO for Windows
resulted in compilation-time decreases and small performance regressions
on a few benchmarks. The changes intended to remove explicit enablement
of cross-language LTO for all builds, but rely on shippable builds being
built with PGO and moz.configure's clever defaulting of cross-language
LTO for PGO'd builds on Windows, which would then enable cross-language
LTO for only shippable builds.
Obviously something went wrong with those changes.
The problem was our defaulting wasn't visible to moz.configure's logic
for how to pass command-line options to the JS subconfigure. We set the
value (`cross`) after the value for `--enable-lto` has been determined,
and the default value is off (that is, `--disable-lto`). Since
moz.configure is very thorough in passing configure options down into
JS, it dutifully looked at what the default value of `--enable-lto` was
supposed to be, and passed `--disable-lto` to JS's configure.
There's some evidence that we knew our defaulting was a little sketchy:
we'd only attempt cross-language LTO when we were performing the PGO use
phase, and only if the value of `--enable-lto` wasn't explicitly passed.
Which was a fine idea--you don't want to override what the user was
trying to do--but in the case of JS backfired on us: the value *was*
coming from the explicitly-passed command-line option of
`--disable-lto`. So JS didn't enable any kind of LTO, with attendant
consequences.
This problem *didn't* happen before the aforementioned change because we
were explicitly specifying that `--enable-lto=cross` should be passed in
the mozconfig, which ensured that the correct setting was passed into
JS. We were just setting `--enable-lto=cross` for *all* builds, which
was less than desirable.
The easiest way to fix all this is simply to put the
`--enable-lto=cross` setting in the Windows mozconfigs, conditional on
`MOZ_PGO_PROFILE_USE`. That placement captures the intent of the
previous attempt at defaulting, but without the troubles described
above: the option explicitly appears on the command line, and
moz.configure will correctly pass it through to the JS subconfigure.
This also makes our Windows configuration closer to our Linux
configuration (the Linux configuration enables cross-language LTO for
both PGO phases, which is arguably a bug).
Differential Revision: https://phabricator.services.mozilla.com/D41080
--HG--
extra : moz-landing-system : lando
This patch makes BuildEnvironmentNotFoundException a subclass of AttributeError as well, because hasattr in
python3 no longer catches all tracebacks but only AttributeError, and we use both hasattr and
BuildEnvironmentNotFoundException to guard against a handful of buildconfig variables in a few places
where it is OK to not have a buildenvironment.
We also use universal_newlines in real_host in init.configure (since I found
that fix while working on the AttributeError one) so that we get the right string type back from the process call
Lastly this patch also uses BytesIO for calling into a ReducedConfigureSandbox as its stdout and stderr pipes,
This ensures that related code handling the sandbox doesn't complain about getbuffer() missing in StringIO in py3.
Differential Revision: https://phabricator.services.mozilla.com/D36605
--HG--
extra : moz-landing-system : lando
What this means is that the sources for clang/llvm are downloaded
separately from the toolchain build (which also means we finally only
download a given version of clang once for all platforms).
In turn, this means the build-clang.py script needs to start with an
existing llvm-project tree, and we choose to make build-clang.py expect
that it's run from the llvm-project root directory.
This also means we don't need to download git for the windows toolchain
task.
Differential Revision: https://phabricator.services.mozilla.com/D40402
By default, the linker chooses a "generic" 32-bit CPU to optimize for,
and LLVM's "generic" 32-bit CPU model doesn't include some features that
are helpful for performance on microbenchmarks. We explicitly specify a
CPU model to ensure the model we want is selected.
On x86-64, we explicitly force a generically good processor model, even
though the automatically selected one didn't seem to hurt benchmarks.
Differential Revision: https://phabricator.services.mozilla.com/D40479
--HG--
extra : moz-landing-system : lando
Bug 1546136 wrapped subprocess execution output to capture cmake's, but
at the detriment of other output, and hiding everything unless an error
occurs.
So instead, we only capture the output when the called process is cmake,
and even when it is cmake, we don't pipe stderr at all (since we only
care about cmake's stdout) and we print out stdout as it comes in rather
than later. We then later check the output for hints at the more useful
cmake logs and dump them.
While here, add verbosity to ninja output (which gives the command
lines, rather than generic "Building foo.o" output).
Differential Revision: https://phabricator.services.mozilla.com/D40142
This change gives us the ability to selectively turn on cross-language
PGO, just like we have the ability to selectively turn on cross-language LTO.
There is room for things to go wrong here: it's not guaranteed that
`--enable-profile-generate=cross` will always be used with
`--enable-profile-use=cross`. Nothing bad will happen in the sense that
the build will succeed, but it's possible that we miss out on
optimizations on the Rust side. Either we fail to generate profile data
for the Rust code, or the Rust compiler fails to use the profile data.
In the future, we may want to default to cross-language PGO to avoid
these kind of mismatches.
Differential Revision: https://phabricator.services.mozilla.com/D39727
--HG--
extra : moz-landing-system : lando
To do properly checks on LLVM version correspondence between `clang` and
`rustc`, we need information about both of those compilers to be
available. The current placement of the LTO/PGO checks is after we know
something about `clang`, but before we know something about `rustc`.
Therefore we need to move those checks after we've gathered information
about `rustc`.
The PGO bits come along for this bug because the LTO bits depend on
them, and we're going to need the Rust information for cross-language
PGO checks in a different bug. So we might as well move everything all
at once.
Differential Revision: https://phabricator.services.mozilla.com/D39390
--HG--
rename : build/moz.configure/toolchain.configure => build/moz.configure/lto-pgo.configure
extra : moz-landing-system : lando
This change will eventually enable us to cross-check `rustc`'s version
with `clang`'s version when doing cross-language LTO/PGO and avoid
people running into peculiar errors at link time.
Differential Revision: https://phabricator.services.mozilla.com/D39388
--HG--
extra : moz-landing-system : lando
Plenty of places use `nproc`, and only a couple use `getconf
_NPROCESSORS_ONLN`. Use the former instead of the latter.
Differential Revision: https://phabricator.services.mozilla.com/D39999
--HG--
extra : moz-landing-system : lando
On Windows in Python 2, the subprocess module requires the use of bytes with
the 'env' argument. For that reason, we would sometimes use byte strings with
'os.environ' like so:
os.environ[b"FOO"] = b"bar"
However, this is a failure with Python 3 as 'os.environ' must only be used with
the text type. This patch creates a new 'setenv' helper that ensures we create
new environment with 'bytes' on Python 2, and 'text' on Python 3.
Differential Revision: https://phabricator.services.mozilla.com/D38237
--HG--
extra : moz-landing-system : lando
These tests are based on `moz_global_class` and `moz_trivial_ctor_dtor` tests,
respectively, but adapted for the semantics of the new attributes.
Differential Revision: https://phabricator.services.mozilla.com/D39718
--HG--
rename : build/clang-plugin/tests/TestGlobalClass.cpp => build/clang-plugin/tests/TestStaticLocalClass.cpp
rename : build/clang-plugin/tests/TestTrivialCtorDtor.cpp => build/clang-plugin/tests/TestTrivialDtor.cpp
extra : moz-landing-system : lando
This patch is in support of adding a variant of Static{Auto,Ref}Ptr for use as
static locals, taking advantage of C++11 "magic statics" such that we can lazily
initialize those variables in a thread-safe way.
In support of those classes, this patch adds two new attributes:
* `moz_static_local_class` to ensure that any instantiations of that class only
occur as static local variables;
* `moz_trivial_dtor` to ensure that these classes do not implicitly call `atexit`
and add a whole bunch of shutdown crap.
`moz_static_local_class` works similarly to `moz_global_class`, except that its
object must only instantiate as static locals.
`TrivialDtorChecker` is based on `TrivialCtorDtorChecker`, with the ctor-specific
bits removed.
Differential Revision: https://phabricator.services.mozilla.com/D39717
--HG--
rename : build/clang-plugin/TrivialCtorDtorChecker.cpp => build/clang-plugin/TrivialDtorChecker.cpp
rename : build/clang-plugin/TrivialCtorDtorChecker.h => build/clang-plugin/TrivialDtorChecker.h
extra : moz-landing-system : lando
Toolchain path for Windows version is `<NDK ROOT>/toolchains/llvm/prebuilt/windows-x86_64` etc, so it isn't '`winnt`.
So we has to replace `host.kernel.lower()` with `windows`.
Differential Revision: https://phabricator.services.mozilla.com/D39474
--HG--
extra : moz-landing-system : lando
...rather than people running into peculiar crashes running their tests
because functions are pointing at the wrong thing.
It would be more robust to version-check `ld`, but I figure people
wanting to do local cross-language LTO builds is rare enough that
setting an environment variable and rerunning configure is not a huge
hardship.
Differential Revision: https://phabricator.services.mozilla.com/D36742
--HG--
extra : moz-landing-system : lando
...rather than people running into peculiar crashes running their tests
because functions are pointing at the wrong thing.
It would be more robust to version-check `ld`, but I figure people
wanting to do local cross-language LTO builds is rare enough that
setting an environment variable and rerunning configure is not a huge
hardship.
Differential Revision: https://phabricator.services.mozilla.com/D36742
--HG--
extra : moz-landing-system : lando
In browsertime.zip we should have:
browsertime/
package.json
package-lock.json
node_modules/
.bin/
browsertime -> ../browsertime/bin/browsertime.js
browsertime/
...
The idea is that we'll fetch browsertime.zip in a generic-worker
environment and be able to run Node.js from within the top level
browsertime/ directory.
Differential Revision: https://phabricator.services.mozilla.com/D38773
--HG--
extra : moz-landing-system : lando
This build re-uses the PGO profile from the win64 build in the
win64-aarch64-shippable-no-eme part of the aarch64 build. Even though
the profile isn't generated on the smae platform, we still get enough of
a performance win to make this worthwhile.
Note that the pgo_flags() in configure need to be tweaked slightly since
we don't supprt the -fprofile-generate flag for aarch64 (we don't build
the clang_rt.profile lib there). So we always want to return the flags
namespace to make sure we get the use_* versions of flags, which we do
need.
Differential Revision: https://phabricator.services.mozilla.com/D38928
--HG--
extra : moz-landing-system : lando
There is less incentive to keep things building with older versions of
clang for OSX builds, and we're going to require an objective-C feature
that was added in clang 5.
Differential Revision: https://phabricator.services.mozilla.com/D38581
--HG--
extra : moz-landing-system : lando
Andreas Tolfsen
Mike Hommey
Ciure Andrei
Nathan Froyd
Sylvestre Ledru
Nicholas Nethercote
André Bargull
Brendan Dahl
Justin Wood
Eric Rahm
Bogdan Tara
Razvan Maries
Andrew Halberstadt
Aaron Klotz
Makoto Kato
Andreea Pavel
Nick Alexander
Mike Shal
Simon Sapin
Agi Sferro