There are a lot of choices and moving pieces in this commit. I elected
to include the mechanics and the target use case in the same commit so
that readers can compare and contrast the implementation and final
expression in one review window.
- Initially, I wanted to make the {AB_CD} substitutions in
LOCALIZED_FILES and not in LOCALIZED_GENERATED_FILES. However, I ran
into conceptual blockers doing this. Fundamentally, LOCALIZED_FILES
is FINAL_TARGET_FILES, and my use case should _not_ be putting files
anywhere near dist/bin. In addition, LOCALIZED_FILES
(FINAL_TARGET_FILES) is handled using manifests, which would need to
grow locale-aware functionality to handle this. That's not desirable.
In addition, if we use manifests, then we lose the powerful locality
of |mach build mobile/android{/base}| re-generating changed
locale-dependent resources. This is similar to how the build system
plumbs dist/idl manifest processing throughout the build: we're
repairing local workflows after moving work into a global process.
For these reasons, this doesn't support {AB_CD} in LOCALIZED_FILES.
- There is even another layer of complexity! There are two axes
involved with these files: AB_CD controls localization and the Make
target controls destination. For the record, it is:
regular builds - AB_CD unset
multi-locale builds - AB_CD set
single-locale repacks - AB_CD set
For the record, the existing logic (before any changes) is:
regular builds - Make target is `libs` in mobile/android/base/locales
multi-locale builds - Make target is `chrome-%` in mobile/android/base/locales
single-locale repacks - Make target is `libs` in mobile/android/base/locales
This commit adds targets for both destinations, and uses Make
chrome-%:: and libs:: magic to control what is invoked in the various
situations. Tricky!
- I added MERGE_RELATIVE_FILES in order to be able to follow-up this
patch with more patches that will get rid of
m/a/base/locales/{moz.build,Makefile.in} altogether, and fold this work
into m/a/base. As it stands, we're already reaching from
m/a/base/locales all the way out to
mobile/locales/.../region.properties, so the existing code doesn't
follow the layout expected between mozilla-central and
l10n-central/$(AB_CD). But that'll impedance will get worse as we
improve the build system dependencies, not better, so we should grow
support for localized resources that aren't exactly as expected.
- I chose to follow Python's syntax for string substitutions. I
would have preferred to mark files that should be localized with a
leading '%'... but I took that for filesystem absolute paths in
moz.build files already. I also considered @AB_CD@ to echo the
preprocessor, but didn't want to open the door to an expecation that
_all_ preprocessor DEFINEs will work in the way {AB_CD} does.
- The generate_*py script changes required a bit of a hack to "turn
off" locale dependent resources. This would have been nicer if we had
marked localized resources with '%'... but we didn't. See the
--fallback flag. The real reason this is needed is that we're doing
work which is more like the work of compare-locales (merging
locale-dependent resources) at build-time rather than repack time. I
don't know why that's the case -- probably when we (I) implemented it,
compare-locales and the whole l10n process was entirely opaque. It's
not worth changing it now, so we use this --fallback flag approach.
- I didn't get to tup support. This should gently fail without
breaking tup builds: any {AB_CD} substitutions just won't be
expanded. I haven't a clue how this should work in tup in the future
(or, more generally, how to make any sense of repacks without
declaring the full set of expected locales at configure time.)
- strings.xml can't be a LOCALIZED_PP_FILES, since we need to
customize the output location based on AB_rCD, and since we need a
little more flexibility than PP_FILES gives for our inputs.
MozReview-Commit-ID: MyfIkNSEzt
--HG--
rename : python/mozbuild/mozbuild/test/backend/data/localized-generated-files/en-US/localized-input => python/mozbuild/mozbuild/test/backend/data/localized-generated-files-AB_CD/en-US/localized-input
rename : python/mozbuild/mozbuild/test/backend/data/localized-generated-files/foo-data => python/mozbuild/mozbuild/test/backend/data/localized-generated-files-AB_CD/foo-data
rename : python/mozbuild/mozbuild/test/backend/data/localized-generated-files/generate-foo.py => python/mozbuild/mozbuild/test/backend/data/localized-generated-files-AB_CD/generate-foo.py
rename : python/mozbuild/mozbuild/test/backend/data/localized-generated-files/en-US/localized-input => python/mozbuild/mozbuild/test/backend/data/localized-generated-files-AB_CD/inner/locales/en-US/localized-input
rename : python/mozbuild/mozbuild/test/backend/data/localized-generated-files/moz.build => python/mozbuild/mozbuild/test/backend/data/localized-generated-files-AB_CD/moz.build
rename : python/mozbuild/mozbuild/test/backend/data/localized-generated-files/non-localized-input => python/mozbuild/mozbuild/test/backend/data/localized-generated-files-AB_CD/non-localized-input
extra : rebase_source : 816b6f220758f2bb3bdd3ec81a2cb02269c6de5b
I wanted to lift this next to the definition of AB_CD, but that
doesn't allow to use it in a backend.mk file, due to the order in
which Makefile, config.mk, rules.mk, and backend.mk are processed.
Therefore, I've put it in a tiny include file, so that it can be used
by a Makefile and a backend.mk file.
This allows the `RecursiveMake` backend to owning defining AB_rCD in
backend.mk files, while not requiring consumers to arrange for AB_rCD
in a sibling Makefile.in file.
Other build backends will need to arrange for AB_rCD themselves: see
following commits.
MozReview-Commit-ID: I7GIzRbCCtf
--HG--
extra : rebase_source : 3277fedb43bc3d8007287c223554a085dae2f198
extra : source : 854c0f43a1f74b4e22aa7638b407580240c90dd5
The last APK produced using the ANDROID_APK_* moz.build/Makefile.in
mechanism was Robocop, so we can get rid of these now.
MozReview-Commit-ID: 9b08ZvvOAoC
--HG--
extra : rebase_source : ac4fea057bf6e731b0f26a1b6902f17a7362076d
Far more files #include "util/Windows.h" than <psapi.h>, so this makes the
style-checker check more things (and makes --fixup mode fix more things).
--HG--
extra : rebase_source : a0c4edf10ad7dd4b5e0265c49035acf242197e6c
This is ancient and the team that used it (gfx) is no longer using it.
MozReview-Commit-ID: HrDgmAU9QeW
--HG--
extra : rebase_source : c4a64965c4ae1a50888893e881a6e8a9688a58b6
This is a new issue that gets linted with flake8 3.5.0. Basically you should
never use a blank except: statement.
This will catch all exceptions, including KeyboardInterrupt and SystemExit
(which is likely not intended). If a catch all is needed, use
`except: Exception`. If you *really* mean to also catch KeyboardInterrupt et
al, use `except: BaseException`.
Of course, being specific is often better than a catch all.
MozReview-Commit-ID: FKx80MLO4RN
--HG--
extra : rebase_source : 7c74a7d0d81f2c984b47aff3a0ee3448b791177b
Now that writing the .purgecaches sentinel is done by |mach build|, we
can remove it from Make. In addition, we can cull the now-unused
app-rules.mk \o/
MozReview-Commit-ID: 6CnAqLeZwzB
--HG--
extra : rebase_source : 2a5b67e723f8e1654627e6e6f26704f6a4e57706
I very much doubt these are used, but even if we are -- we shouldn't
support this type of local customization, since it doesn't extend to
non-Make-based backends.
With the customization point removed, there's no way to set ETAGS, so
we remove what little support there was for generating Emacs tags.
MozReview-Commit-ID: IEF2Q4tISEn
--HG--
extra : rebase_source : 3bc8e651c03517edb797032db6ce60ed8852d9fa
Now that writing the .purgecaches sentinel is done by |mach build|, we
can remove it from Make. In addition, we can cull the now-unused
app-rules.mk \o/
MozReview-Commit-ID: 6CnAqLeZwzB
--HG--
extra : rebase_source : 7427ca9aecc6619f48090c6a26c9a2e0e7d37bad
I very much doubt these are used, but even if we are -- we shouldn't
support this type of local customization, since it doesn't extend to
non-Make-based backends.
With the customization point removed, there's no way to set ETAGS, so
we remove what little support there was for generating Emacs tags.
MozReview-Commit-ID: IEF2Q4tISEn
--HG--
extra : rebase_source : 140e07f78dbd72391c70c48f25e2b85c924fe78d
Fix several problems when building Breakpad with new NDK unified
headers.
- Unified headers define its own tgkill wrapper, so rename our own
wrapper to __tgkill.
- Unified headers define user_fpxregs_struct for all API levels, so
don't redefine it.
- Only the target sources under google-breakpad/src/common/linux should
use custom Android headers, so change the includes line in moz.build to
use OS_INCLUDES.
MozReview-Commit-ID: HGnUMu5vDUM
--HG--
extra : rebase_source : f5c29c9949a48a8376f84dcc676e5f8df886f130
Fix several problems when building Breakpad with new NDK unified
headers.
- Unified headers define its own tgkill wrapper, so rename our own
wrapper to __tgkill.
- Unified headers define user_fpxregs_struct for all API levels, so
don't redefine it.
- Only the target sources under google-breakpad/src/common/linux should
use custom Android headers, so change the includes line in moz.build to
use OS_INCLUDES.
MozReview-Commit-ID: HGnUMu5vDUM
--HG--
extra : rebase_source : 07da575044ca805a3f7f8ba87be8a8c279f17379
This patch adds basic support for the fuzzing interface in the JS engine on top
of the last patch. This includes all the necessary code except for actual
targets (just an example target skeleton) and also makes sure that the fuzzing
code is packaged for the standalone release.
MozReview-Commit-ID: D6Tyebz3jZS
--HG--
extra : rebase_source : f07a5d2bf8e09fb6c93501e58fbb958004e2fa05
This patch adds basic support for the fuzzing interface in the JS engine on top
of the last patch. This includes all the necessary code except for actual
targets (just an example target skeleton) and also makes sure that the fuzzing
code is packaged for the standalone release.
MozReview-Commit-ID: D6Tyebz3jZS
--HG--
extra : rebase_source : 58e4d85e657347b061de0ed912365f2a955a86e3
This patch disables the stdcxx-compat check for the sm-fuzzing build which
requires patching autospider as well. Furthermore, it switches the build
to linux64-clang-6-pre because the older clang 3.9 does not support trace-pc
instrumentation. Finally, it excludes fuzzing parts from the vanilla allocation
check.
MozReview-Commit-ID: FdhCIFdUore
--HG--
extra : rebase_source : c41bda01cb42f2ef0cd5a1675d88bdb55d9dc8c9
The MSVC linker winds up generating import libraries when linking some of
our executables, presumably because they contain functions that are
__declspec(dllexport). By default the import libraries get written
alongside the exe, so we force them to be written to the objdir so they don't
clutter up dist/bin.
MozReview-Commit-ID: 7DTfCo3OdDQ
--HG--
extra : rebase_source : fea69e8f60633b824726269c2296af9fe812d3ed
Historically we built all our binaries in directories in the objdir, then
symlinked them into dist/bin. Some binaries needed to be copied instead
so that certain relative path lookups work properly, so we resorted to
sprinkling `NSDISTMODE=copy` around Makefiles.
This change makes it so we build PROGRAMs (not any other sort of targets)
directly in dist/bin instead. We could do the same for our other targets
with a little more work.
There were several places in the tree that were copying built binaries to
some other place and needed fixup to match the new location of binaries.
On Windows pdb files are left in the objdir where the program was
originally linked. symbolstore.py needs to locate the pdb file both to
determine whether it should dump symbols for a binary and also to copy
the pdb file into the symbol package. We fix this by simply looking for
the pdb file in the current working directory if it isn't present next
to the binary, which matches how we invoke symbolstore.py.
MozReview-Commit-ID: 8TOD1uTXD5e
--HG--
extra : rebase_source : 9140be949b206bb595d9188ce7e8357347ecd9a9
Work around excessive command-line length issues by
disabling incremental rust compilation, which is enabled
by default outside `cargo --release` starting with Rust 1.24.
Incremental rust builds shouldn't help much in automation,
where sccache provides the only continuity between build
environments. In the meantime, they add a lot of object
files to the link line.
See https://github.com/rust-lang/rust/pull/47507 about addressing
the underlying issue upstream.
MozReview-Commit-ID: LRwUj3fhiaO
--HG--
extra : rebase_source : 1739a7570b2e7fe40ead3b301ea20c2fe79f0431
The pytest cache plugin writes its cache in the srcdir, which means that it
shows up in `hg status`, which is annoying. Writing files to the srcdir is
generally bad practice anyway, so we disable this plugin to stop this
from happening.
MozReview-Commit-ID: HytLLMUtKlc
--HG--
extra : rebase_source : f6acbf3650881312cef051126387220a0f78597f
Back when I filed this bug, $appname-$version was already useless, but
there actually were multiple supported versions (e.g. 3.5/3.6). So it
made sense to have e.g. firefox-3.5 and firefox-3.6, but not
firefox-3.5.12 and firefox-3.5.13.
Fast forward 10 years, and we change "major" versions every 6 to 8
weeks, have multiple chemspills every other week, and installing to
firefox-57.0, firefox-57.0.1, firefox-57.0.2, firefox-57.0.3 doesn't
make any kind of sense. Even firefox-56, firefox-57 is pretty much
useless.
There /kind/ of was some usefulness to the version in the SDK
directories, but those are gone.
I'm pretty sure no downstream is actually using versioned directories
anyways.
At this point, it seems better to just use the application name, without
the version. A case could be made about ESR, but that would be better
handled with a separate application name (e.g. firefox-esr).
--HG--
extra : rebase_source : 333e10ea1316714bf0008ec772b35093edfc45ff
Some of the variants are multi-platforms, and having
--enable-stdcxx-compat in their definition will break when the option is
moved over to python configure.
While here, prepare for --enable-stdcxx-compat actually doing something
(it currently doesn't), by adding an exception for it in
check_vanilla_allocations.py.
--HG--
extra : rebase_source : ee1647421542209cf0137db703c4f7e7f06cbc91
This has the virtue of not executing python three times during configure
just to read the same value of milestone.txt and munge it. We can also
remove milestone.py as a happy side effect, so all the milestone
computations can be done in init.configure.
We only had this "for flexibility with other platforms", but given that
we set it to the same thing for all platforms, and nobody has tried to
change that, this flexibility isn't actually needed.
Bug 1256642 introduced magic at the emitter level to determine whether a
binary contains C++ sources and should be linked with the C compiler or
the C++ compiler.
Unfortunately, the Binary() moz.build template always adds C++ OS
libraries on Android (through STLPORT_LIBS), and C++ libraries on Linux
(stdc++compat).
The latter only ends up forcing every Binary() to be linked with the C++
linker, which is unfortunate, but doesn't cause much problems. The
former, however, involving OS libraries, the magic from bug 1256642
doesn't kick in, so we end up trying to link C++ OS libraries with the C
linker. Which ends up failing, because the libraries in STLPORT_LIBS
require -lm, which, while it's added by the C++ compiler when linking,
is not when the linkage is driven by the C compiler.
Because the fallible library, linked to all GeckoBinary()s is a C++
library, we still ended up linking with the C++ compiler on Android, so
this wasn't actually causing any problem... until I tried to remove that
fallible library in bug 1423803.
Anyways, the core problem is that moz.build evaluation is happening too
early to know whether any C++ sources are being linked together, so
there is no way the Binary() template can do the right thing. So this
change moves the logic to the emitter.
This also changes the type of STLPORT_LIBS to a list.
--HG--
extra : rebase_source : a70ddf7a132f94dc10e7e1db94ae80fb8d7a269f
Back when mozalloc was a separate library, the xpcom glue code could not
use the infallible allocator API. But since bug 868814, that's not the
case anymore, so we can safely include mozalloc.h when XPCOM_GLUE is
set.
--HG--
extra : rebase_source : a8fbf8dc7020765d7287e2eb7ceaf41c99be8b18
This accounts for LOCAL_INCLUDES in the moz.build files, as well as the
default INCLUDES specified in config.mk that are used for host
compilation. Since some of the HOST_CFLAGS were also used for linking,
those flags are split off into HOST_C{XX}_LDFLAGS so that
the linker-only flags can be placed in those variables.
MozReview-Commit-ID: J1LxIZVeFJ
--HG--
extra : rebase_source : ed7293604e5428e3124f1ecfb2b706e087436b72
These flags were added way back in bug 59454 to both the compiling and
linking rules for HOST_SIMPLE_PROGRAMS. However, the INCLUDES aren't
actually needed when linking so we can safely remove them.
MozReview-Commit-ID: 8QywO7tGPpU
--HG--
extra : rebase_source : 34870adcf73b459771413e52bc84c6c63669d0f9
Both SFLAGS and ASFLAGS are used to compile assembly, but SFLAGS include
DEFINES and LOCAL_INCLUDES whereas ASFLAGS do not. It seems easiest to
just separate them into two different ComputedFlags values so that the
backend can distinguish between the two types.
MozReview-Commit-ID: Bkm3621ImJG
--HG--
extra : rebase_source : 420204e37d591512f700d77b780939d20c2feeb0
These have been replaced by stl-headers.mozbuild and
system-headers.mozbuild
MozReview-Commit-ID: EpPasSzgiJk
--HG--
extra : source : 86302df0b38a05ce8f6f830e324631a42c1c643e
The make-system-wrappers.py invocation is largely identical to
make-stl-wrappers.py, though this script generates wrappers for both the
STL headers and every other system header that can be used.
Note that the nsprpub script didn't create multiple layers of
subdirectories properly, so for example the 'ia64/sys/inline.h' wrapper
is now generated properly. Additionally, MOZ_SYSTEM_ICU define was
incorrectly using '#ifdef' instead of '#if ... == 1', which causes those
unicode headers to have wrappers when they shouldn't. These will show up
as differences when comparing the Makefile output to the moz.build
output.
MozReview-Commit-ID: KvQAawfzXao
--HG--
extra : source : 5a967cc85e28e63c283a81e2c76444a76dfbd266
This is fairly straightforward to represent as a GENERATED_FILES, though
we have to take some care to construct the outputs tuple correctly. This
script needs to run during export, and unfortunately none of the STL
headers have proper file extensions, so the 'new' header is
special-cased in the recursive make backend to serve as a marker for
running it in the correct tier.
We can't remove the stl-headers file yet because it is still used for
the system header generation.
MozReview-Commit-ID: 3tQTOY0LAsQ
--HG--
extra : source : 828d43ec1b16edaac69c42f15561f26e209051f1
These don't actually install correctly, because 'private' is already a
system wrapper. Neither of these files are included anywhere, so there's
no harm in removing them.
MozReview-Commit-ID: 1IAFMZDvKDV
--HG--
extra : source : 8d770908a5b9a3421294c93f8a34a14265674e9b
This is no longer used in the tree, and the $ will get eaten somewhere
along the way on the path from moz.build to the python invocation by
make despite being shell_quoted.
MozReview-Commit-ID: 5mwWQYDn8Nf
--HG--
extra : source : ffe9aed0944f57e329ee279f5b14266d5c4069a2
These have been replaced by stl-headers.mozbuild and
system-headers.mozbuild
MozReview-Commit-ID: EpPasSzgiJk
--HG--
extra : rebase_source : 33bc7a5a8de794f882bd1438ac800da2150a2ce6
The make-system-wrappers.py invocation is largely identical to
make-stl-wrappers.py, though this script generates wrappers for both the
STL headers and every other system header that can be used.
Note that the nsprpub script didn't create multiple layers of
subdirectories properly, so for example the 'ia64/sys/inline.h' wrapper
is now generated properly. Additionally, MOZ_SYSTEM_ICU define was
incorrectly using '#ifdef' instead of '#if ... == 1', which causes those
unicode headers to have wrappers when they shouldn't. These will show up
as differences when comparing the Makefile output to the moz.build
output.
MozReview-Commit-ID: KvQAawfzXao
--HG--
extra : rebase_source : 758e325c6ec192d83fcfa10f8c878ba2629e45a3
This is fairly straightforward to represent as a GENERATED_FILES, though
we have to take some care to construct the outputs tuple correctly. This
script needs to run during export, and unfortunately none of the STL
headers have proper file extensions, so the 'new' header is
special-cased in the recursive make backend to serve as a marker for
running it in the correct tier.
We can't remove the stl-headers file yet because it is still used for
the system header generation.
MozReview-Commit-ID: 3tQTOY0LAsQ
--HG--
extra : rebase_source : bd9f00e45a7bce4daaa0e1c16e22b28536658e37
These don't actually install correctly, because 'private' is already a
system wrapper. Neither of these files are included anywhere, so there's
no harm in removing them.
MozReview-Commit-ID: 1IAFMZDvKDV
--HG--
extra : rebase_source : 6e153324d2add32a179d07020270db96e7ec6dd0
This is no longer used in the tree, and the $ will get eaten somewhere
along the way on the path from moz.build to the python invocation by
make despite being shell_quoted.
MozReview-Commit-ID: 5mwWQYDn8Nf
--HG--
extra : rebase_source : f2289f5f5e15529475cac6cba3fc675592c111f2
ffxbld
Nathan Froyd
Philipp Kewisch
Nick Alexander
arthur.iakab
Kim Moir
Sebastian Hengst
Jason Orendorff
Dave Hunt
Gurzau Raul
Jeff Walden
Andrew McCreight
Andrew Halberstadt
André Bargull
Coroiu Cristina
Jim Chen
Cosmin Sabou
Narcis Beleuzu
Matt Howell
Nika Layzell
Christian Holler
Ted Mielczarek
Simon Fraser
Ralph Giles
Csoregi Natalia
Sylvestre Ledru
Dragana Damjanovic
Tom Prince
Mike Hommey
David Major
Ting-Yu Chou
Andreea Pavel
Mike Shal
Alfredo.Yang
Brindusan Cristian
Chris Manchester
Ryan VanderMeulen
Phil Ringnalda
Jonathan Watt