There is a known compilation bug that is fixed in Visual Studio 2017
15.3.
This commit establishes minimum version requirements for Visual
Studio 2017 to ensure modern and more supported toolchains are used.
We previously only had a minimum version requirement for Visual
Studio 2015.
MozReview-Commit-ID: WRrry8J5Yf
--HG--
extra : rebase_source : 1e60bb4277ae1ec679cfa103b0a730a7260b52c2
Technically this turns on gnu++14. I encountered a few errors when using c++14:
1) _USE_MATH_DEFINES needed to be defined for MinGW
2) MinGW did not define _finite under c++14
3) MinGW's float.h did not define Microsoft specific float functions under c++14
All of these were because c++14 defines _STRICT_ANSI_ which MinGW obeys and
avoids defining certain functions. The first two could be patched around, but
the third was a blocker, so we switched to gnu++14
MozReview-Commit-ID: 6Y7gEQgApYp
--HG--
extra : rebase_source : dabbd40c049c36e780b585e0bef0a8e25887d089
This is needed so we can import the WPT manifest parser.
Like for the reftest parser, this should only be temporary until we no
longer parse test manifests as part of Files() evaluation.
MozReview-Commit-ID: 2dmUfP8tLd2
--HG--
extra : rebase_source : 36624618f96fc89bae88172947b21108740d841d
Some moz.build files need to import in-tree Python modules in order to
parse test manifests. While bug 1402010 details a better solution
to remove this requirement, the easy fix is to expand the sparse
checkout to include the missing files.
MozReview-Commit-ID: FSx6u6r2XfE
--HG--
extra : rebase_source : 6e406dd76b4f78521cbbdc40110d0610c75e2b08
extra : amend_source : f68341a14f50c8ff6e7b62214fa455b75eabe908
Vendor new dependency for compare-locales, python-fluent.
This is the 0.4.2 release of python-fluent.
Also, make mach command a bare minimum wrapper.
The compare-locales mach command used to own a couple of defaults
and opinions.
Now that those opinions and defaults are in the Makefiles, this
command can be much simpler. As a side effect, this should
make the thunderbird port easier, where none of the mach defaults
worked.
Update l10n.mk for compare-locales 2.x:
The command line interface is different now, with positional arguments.
Also, the l10n config file isn't a default anymore (that never worked
for thunderbird).
And the merge dir is now the parent of the locale dir, much like
we already anticipated in the code.
MozReview-Commit-ID: DxVVKyVSt5y
--HG--
extra : rebase_source : 7c591fea8c7d9ecef615fb56f9359d4f1bd4b340
Strictly speaking we don't need all files in the directories listed
in the profile. But the checkout is still small enough and it is far
less effort than cherry-picking every file needed by every toolchain
task.
This brings the checkout down to ~3700 files, which only takes 1-2s.
MozReview-Commit-ID: 2BpKdZ2Pvx9
--HG--
extra : rebase_source : e9589b29f7e1b60ac7ee4e8050689dc3d5a8f418
extra : source : 2c0d6e566cc719823515ec2403e1d6b2ace955aa
toolchain.configure checks for information about the compilers we're
using and accumulates additional flags that we might need to
pass (e.g. switches for C/C++ versions, proper compiler targets, etc.),
and then rechecks the compilers with those additional flags to verify
that those flags are sufficient to configure the compiler properly.
Only after we've checked for the proper flags do we move on to verifying
the compiler versions are sufficient.
It's possible, however, that the flags we try to add might not be
supported by the compiler being tested, in which case we'd get an
unfriendly error about an "unknown compiler", rather than something like
"version X of Y not supported". In this case, we'd rather use the
information we gathered from the first run to provide information
messages. So we modify the second check to ignore any thrown exceptions
during the check itself.
This change results in another problem: the check for whether we had any
extraneous flags is done prior to version checks for the compilers we
support, which is also unhelpful. We choose to move this check after
the version checks themselves.
gcc 4.9 is the last version in Android NDK and our minimum requirement of gcc is 4.9+. --with-android-gnu-compiler-version is unnecessary option because gcc version of Android is always 4.9.
MozReview-Commit-ID: 1sutqlvbQU5
--HG--
extra : rebase_source : 77e42aba81da5276bcddd945ea41895ce2461afa
This adds some new optimization strategies. For tests, we use Either(SETA,
SkipUnlessSchedules), thereby giving both mechanisms a chance to skip tasks. On
try, SETA is omitted.
MozReview-Commit-ID: GL4tlwyeBa6
--HG--
extra : rebase_source : f208e4960cf76a9dfe77634b87f0058f676e9fa9
extra : source : 046d705929f7a41e977eec19c8503afccdec7592
Strictly speaking we don't need all files in the directories listed
in the profile. But the checkout is still small enough and it is far
less effort than cherry-picking every file needed by every toolchain
task.
This brings the checkout down to ~3700 files, which only takes 1-2s.
MozReview-Commit-ID: 2BpKdZ2Pvx9
--HG--
extra : rebase_source : 916c6722c6d495cdccea0076c673d75e4ee48f28
extra : source : 2c0d6e566cc719823515ec2403e1d6b2ace955aa
This adds some new optimization strategies. For tests, we use Either(SETA,
SkipUnlessSchedules), thereby giving both mechanisms a chance to skip tasks. On
try, SETA is omitted.
MozReview-Commit-ID: GL4tlwyeBa6
--HG--
extra : rebase_source : 4cf3efc9c57bb14d2f44147c8881d0a0a18703d6
extra : source : 046d705929f7a41e977eec19c8503afccdec7592
libstdc++ support is broken after moving to moz.configure. No one uses this option and NDK will remove GCC, so we should remove this and --with-android-cxx-stl option.
MozReview-Commit-ID: 3mqyHoRCE00
--HG--
extra : rebase_source : 35aa911a69e159e67f624ab5ab9aea8af4c5342f
LLVM can optimize code to a form `if B && A` when A is
always false but B is undefined. This triggers a valgrind
memcheck warning since the conditional depends on undefined
data but in practice it can never have side-effects.
Rust 1.20.0 seems to trigger this in the Option code. Since
we believe the transform is correct in these cases and
valgrind is incorrect to warn, we surpress the error.
Thanks to Julian Seward for the analysis and help
writing the suppression entries.
MozReview-Commit-ID: pF1Bmy5PRY
This was done in bug 736564 for the xulrunner SDK, which later became
the firefox SDK, which is now gone. So we don't actually need to keep it
separate anymore (except for logalloc/replay, which still needs to link
it directly, so we keep the library definition intact so it can be
referenced ; we just don't DIST_INSTALL it anymore, and always make it
linked into mozglue)
--HG--
extra : rebase_source : e4d0627ec907fe0139df5c0b2b9f7d04b43c7c78
The name `android-gradle-build` is an accident of history; let's rename it
before we attempt major surgery on it.
--HG--
rename : taskcluster/docker/android-gradle-build/Dockerfile => taskcluster/docker/android-build/Dockerfile
rename : taskcluster/docker/android-gradle-build/README.md => taskcluster/docker/android-build/README.md
rename : taskcluster/docker/android-gradle-build/REGISTRY => taskcluster/docker/android-build/REGISTRY
rename : taskcluster/docker/android-gradle-build/VERSION => taskcluster/docker/android-build/VERSION
rename : taskcluster/docker/android-gradle-build/buildprops.json => taskcluster/docker/android-build/buildprops.json
rename : taskcluster/docker/android-gradle-build/dot-config/pip/pip.conf => taskcluster/docker/android-build/dot-config/pip/pip.conf
rename : taskcluster/docker/android-gradle-build/oauth.txt => taskcluster/docker/android-build/oauth.txt
cpu_arch_dir isn't absolute path, we cannot always detect thumb library for c++ library. Our build tool still uses NDK r11c, so we should check thumb library correctly.
MozReview-Commit-ID: 912ZDNeUcKc
--HG--
extra : rebase_source : 8b69348ef3280f32b7a228d66462070413a25e73
The linker detection essentitally invokes `$CC -Wl,--version` to
determine which linker will be used. This invocation works OK for GCC
from the Android NDK, which can find the linker in the same directory;
it doesn't work so well for clang from the NDK, which needs additional
command-line options to point at the directory containing the linker.
We already have extra_toolchain_flags for communicating such options to
other compiler invocations for the target; we should be using
extra_toolchain_flags here as well.
This removes the double-include macro hackery that we use to define two
separate string types (nsAString and nsACString) in favor of a templated
solution.
Annotations for Valgrind and the JS hazard analysis are updated as well as
the rust binding generations for string code.
--HG--
extra : rebase_source : 63ab2c4620cfcd4b764d42d654c82f30f984d016
extra : source : 9115364cd4aa078c49bba7911069f8178e55166f
The Sphinx documentation only needs access to a relatively small number
of files in the repo in order to be generated. It is a good candidate
for using sparse profiles.
This commit defines and uses a "sphinx-docs" sparse profile containing
only the files relevant to Sphinx documentation generation.
There are some quirks with the profile:
* All moz.build files are included. This bloats the profile
by >1000 files. Worse, it realizes directories that have no business
being realized. This clutters the checkout and makes it harder to
find things. There is a moz.build reader that knows how to retrive
file data from Mercurial. We could use that. This feels like follow-up
fodder.
* All mach_commands.py files are included. `mach help` says you can do
things that you aren't able to do in the sparse checkout. There isn't
a good way to add all *.py files while excluding mach_commands.py
files. We /could/ do it with regular expressions. But those are slow.
Let's leave it as is for now and come up with a better solution later.
MozReview-Commit-ID: 7yiqGGE1nAh
--HG--
extra : rebase_source : c148040ea3618e8bfdd369b6f48fc60c6d179285
extra : source : b76e2f6204b20de137f2566dff8121ff3abe5760
To construct an empty set, we need to use the `set()` notation. In order
to do that, we need to expose `set` to the moz.build sandbox.
MozReview-Commit-ID: DMyKnF0FEx2
--HG--
extra : rebase_source : 5cfe8080ec333a1eca70cd3edba2aaaff6406820
Add a BuildFlag annotation, which when specified for classes, will wrap
generated code in `#ifdef` or `#ifndef` blocks. This functionality is
used for conditionally excluding generated code when NIghtly becomes
Beta, without the need to regenerate bindings.
MozReview-Commit-ID: L2NFM8CHKqF
--HG--
extra : rebase_source : 6ebc82d11fd1aa4aeb57a46262e678480d23de83
python-hglib is a Python client for Mercurial's command server. It
facilitates querying Mercurial efficiently (using a single process)
and without having to parse output in the common case.
Let's vendor it so we can make use of it for more advanced Mercurial
scenarios.
Content vendored from changeset 820d7c1e470a without modifications
(other than deleting unwanted files).
As part of vendoring, we add the package to the virtualenv and make
it available to mach.
MozReview-Commit-ID: F4KLbW1lAvk
--HG--
extra : rebase_source : ad81f3aa4843312d300d80ce563d40736af2c16a
This adds some new optimization strategies. For tests, we use Either(SETA,
SkipUnlessSchedules), thereby giving both mechanisms a chance to skip tasks. On
try, SETA is omitted.
MozReview-Commit-ID: GL4tlwyeBa6
--HG--
extra : rebase_source : 0c1ce762afc7a691788379d4f4206df106f6df63
The HTTP server can dynamically map URLs to local filesystem paths.
We employ this so Speedometer's files are available to the server.
MozReview-Commit-ID: EpF1aD6meZH
--HG--
extra : rebase_source : 70f80e073c2bec92c9429dcc35cd12e27018f3cb
Using /home/worker is the build directory has a 30% talos performance
loss, because test machines has a /home mount directory.
MozReview-Commit-ID: 554IPMRWgzK
--HG--
extra : rebase_source : 00827d3f6bd705419bc801eb05b543af1ddc274f
In some cases, they are false positives. In other cases, the callsite is #ifdef-ed
out, so it's a true positive, but not one we would correct.
MozReview-Commit-ID: 6ThZH3wEXTp
--HG--
extra : rebase_source : aabdb93cb924e74cf9592a93c86062c4435ceaad
``print()`` has no business being in library code like this. It was
a holdover from this code being copied from bootstrap. So remove it.
While we're here, replace the generic exception with a specific one.
We don't want to be swallowing bugs via ``except Exception``.
MozReview-Commit-ID: 49goUstfPBz
--HG--
extra : rebase_source : a821159bd12a449ed1a0edf21a1f9eb29711ad95
An inline comment wanted it. The change is trivial.
MozReview-Commit-ID: CqyOYqNzwzr
--HG--
extra : rebase_source : d779d6bbca4aa57d9e95ead3a8403e0163106bdf
The file was using 2 space indent. I ran `autopep8` on it. Most of the
changes are whitespace only.
MozReview-Commit-ID: 1C7yDUQCfKl
--HG--
extra : rebase_source : 9f2b52ef6d0887fbaa7f8410b32348d371e884d8
Mercurial's sparse checkout support allows "profile" files defining
what's in sparse checkouts to be defined in-tree. This convenient
feature means you simply need to point a client at a path inside the
repository and it dynamically resolves what files to include in the
checkout. As you update revisions, the "profile" pulls in updates
to the underlying file.
We introduce 2 sparse profiles: 1 for mach and another for taskgraph.
The goal of the "mach" profile is to provide enough files to run
`mach`. If you activate this profile and run `mach`, it runs without
error. But there are practically no commands available. So it isn't
terribly useful.
The "taskgraph" profile allows us to run `mach taskgraph`. This
profile demonstrates a sparse profile feature: including other
profiles. The "taskgraph" profile is thus a union of "mach" and
its own entries.
There is definitely some fat in these profiles. I didn't feel like
chasing the long tail and getting overly granular with the profiles.
If we want to optimize later, we can do that.
For reference:
Full checkout: ~234,000 files
mach: ~2,000 files
taskgraph: ~3,600 files
MozReview-Commit-ID: 7pALt0MwHfE
--HG--
extra : rebase_source : 1a4ba4b8a63c522dab2841e2c0019501476da2fe
python-hglib is a Python client for Mercurial's command server. It
facilitates querying Mercurial efficiently (using a single process)
and without having to parse output in the common case.
Let's vendor it so we can make use of it for more advanced Mercurial
scenarios.
Content vendored from changeset 820d7c1e470a without modifications
(other than deleting unwanted files).
As part of vendoring, we add the package to the virtualenv and make
it available to mach.
MozReview-Commit-ID: F4KLbW1lAvk
--HG--
extra : rebase_source : 39321a880a13a0b0323a7217f538978b729e2afe
Sparse checkouts may not have all mach_commands.py files.
mach raises an error when a mach_commands.py file is missing.
So, we teach the mach driver to ignore missing file errors when
a sparse checkout is present.
The added code is optimized to avoid an import of mozversioncontrol
and some I/O as part of resolving the repo and VCS binaries because
this file is in the critical path of all mach commands and avoiding
I/O is worthwhile. Since we aren't using sparse checkouts in the
common case, this effectively makes the new code 0 cost.
MozReview-Commit-ID: C6itJga31t5
--HG--
extra : rebase_source : 4b2c18d30ff8b923a940c80ac81372a4076b8fdc
"mach" in this scope is both the mach module and the driver instance.
Let's change the latter's variable name so we can access the mach
module.
MozReview-Commit-ID: Db6sxDFl2oo
--HG--
extra : rebase_source : 0a51c7514cfdba40483b3b9b361cd03b8000c2c9
The Repository interface gains a new method to determine if a sparse
checkout is present. Mercurial's implementation is somewhat crude,
but it should work (Mercurial's sparse support is still experimental
and I only intend to support sparse checkout in Firefox CI until it
is less experimental). Git's always returns False (for now).
To prove it works and to expose the information more widely, we hook
it up to moz.configure. We do this by first implementing a function
that returns a Repository instance. Then we simply call a function on
it to resolve the sparse checkout flag.
MozReview-Commit-ID: AlsT5LdSPdZ
--HG--
extra : rebase_source : f1e9aaa7d15f11c7c5e8d268d4ad82468732103b
This change adds an upload-generated-sources task kind that runs after nightly
builds, fetches their `target.generated-files.tar.gz` artifact, and uploads
all the contained files to an S3 bucket. For actual nightly and release builds
on SCM level 3 trees, the S3 bucket is configured to be publicly accessible,
so that tools like Socorro will be able to fetch generated source files that
appear in crash reports, and debuggers will be able to fetch generated sources
when they show up while debugging Nightly or Release builds.
There are also level-2 and level-1 S3 buckets configured for builds happening
on trees of other levels such as try. They are not configured as publicly
accessible, but they exist so that these tasks can be tested in try.
MozReview-Commit-ID: Js1HRftbtep
--HG--
extra : rebase_source : b1172c9cc8b8be437d3b94a6bf0ff6b2f7d3508b
extra : source : 73bf88110b3821d62a3d393e85b56896a12f2930
This change makes us upload an `$(PKG_BASENAME).generated-files.tar.gz` archive
alongside other build artifacts which contains all the generated source files
from the build. A change after this will introduce an `upload-generated-sources`
task to take this artifact and upload the individual files to an S3 bucket.
This will be used to provide links to generated source files when they appear
in stack traces in crash reports.
MozReview-Commit-ID: 6yQAdlZ5q3O
--HG--
extra : rebase_source : d92fb17ae737d1360e9724997f6688e29bedef12
extra : source : 14d18d7cf454c4c3d0f6d49d1d01660e06e4be4b
A followup change will be adding a new automation step that wants to be skipped
in artifact builds, and this will make that simpler.
MozReview-Commit-ID: 5xwRB9eCRQn
--HG--
extra : rebase_source : 2fccd9d128ab92c98515762a62a0a2e89bf9ca24
extra : source : a02695cbf5762eb0eb7087239319807eb447ca1e
shutil.copy2() will fail if the destination directory doesn't exist.
Switch to copy_tree() instead so we don't need to worry about the
error cases of copy2() and copytree().
MozReview-Commit-ID: 3kHfgL57KfX
--HG--
extra : rebase_source : c7335b0c2854d53699dda0f0d2bd9d17b57c4e5d
We define extra_toolchain_flags for passing extra flags to the target
compiler during configure. But the way things are currently set up, we
pass those flags to the host compiler during configure as well. This
behavior is incorrect, and we should only be passing the flags from
extra_toolchain_flags if we're compiling for the target.
This is similar to what we had until bug 1278456 removed them when we dropped
support for older libstdc++, but for a different symbol.
--HG--
extra : rebase_source : 641fc6c86c8f47e3dbd752bc20056f61646541a7
That the wrapper implementation works has been verified by creating a
dummy program such as:
$ cat test.cc
#include <thread>
int main() {
std::thread([]() {
printf("foo\n");
}).join();
return 0;
}
And compiling it with and without the hack:
$ g++ -fno-rtti -o test test.cc -lpthread
$ objdump -TC test | grep UND.*GLIBCXX_3.4.22
0000000000000000 DF *UND* 0000000000000000 GLIBCXX_3.4.22 std:🧵:_State::~_State()
0000000000000000 DF *UND* 0000000000000000 GLIBCXX_3.4.22 std:🧵:_M_start_thread(std::unique_ptr<std:🧵:_State, std::default_delete<std:🧵:_State> >, void (*)())
$ ./test
foo
$ g++ -fno-rtti -o test test.cc $objdir/build/unix/stdc++compat/stdc++compat.o -lpthread
$ objdump -TC test | grep UND.*GLIBCXX_3.4.22
$ ./test
foo
--HG--
extra : rebase_source : 53ca8e2d0424eaeb539d50510c441c8a3252c819
Sebastian Hengst
Mike Hommey
Gregory Szorc
Tom Ritter
Chris Manchester
Wes Kocher
Geoff Brown
Phil Ringnalda
Axel Hecht
Nathan Froyd
Makoto Kato
Dustin J. Mitchell
Ralph Giles
Nick Alexander
Botond Ballo
Bob Owen
Kartikaya Gupta
Ted Mielczarek
Eric Rahm
Jim Chen
Ryan VanderMeulen
Johan Lorenzo
David Major
Wander Lairson Costa
Philip Chimento
Ting-Yu Chou
Tristan Bourvon
Chris Peterson
Ted Mielczarek
Dave Hunt
Eugen Sawin