This commit adds new moz.build sandbox symbols that are intended to be used
for localized files: LOCALIZED_FILES and LOCALIZED_PP_FILES. They are currently
just do-nothing subclasses of FinalTarget[Preprocessor]Files, but the next
change in this series will add support for them to the recursive make backend.
Because they subclass FinalTarget[Preprocessor]Files, build backends that are
not concerned about localized builds should be able to handle them as if
they were FINAL_TARGET[_PP]_FILES without any additional code.
MozReview-Commit-ID: K0baBZ0F7av
--HG--
extra : rebase_source : 323e2993638fb0ba44ed89a4e0edd16b27a287e0
extra : source : e3ce81bc209b09b6771d7056d1fb06a65e27dc0d
This commit adds support for handling LOCALIZED_FILES and LOCALIZED_PP_FILES
in the recursive make backend. They get special handling because they have
a few special needs:
* They run during the libs tier, so that repacks work.
* The filenames cannot be determined at build-backend generation time,
since repacks run configure once and then repack multiple locales using
the generated backend files, so they are written with to be expanded with
MERGE_FILE by make so that the file gets picked up from the proper locale dir.
Other build backends that aren't trying to handle localized builds will
silently handle these like FINAL_TARGET_FILES, which is fine until we revamp
our l10n repack story.
MozReview-Commit-ID: 2LZhPZNhQ4S
--HG--
extra : rebase_source : dde0eb2e0ffe0bac5e5a6ab2c5f5724c06154101
extra : source : 5ee033a3c356bed86219699698abfe538370740a
In order to determine if we need to re-run configure, we write
a JSON file representing the evaluated mozconfig. If this JSON
file changes, configure (and config.status for that matter) is
out of data and it is re-executed.
This commit moves the generation of that JSON file to Python.
MozReview-Commit-ID: 636rpSY7gOm
--HG--
extra : rebase_source : ee1defd74decfd64ffb66a45b053dada58de04fb
This is a pretty straightforward port of the logic. But we
even go a step further: we delete the file in the objdir if there is
no source mozconfig!
MozReview-Commit-ID: AHFFzy6mXRY
--HG--
extra : rebase_source : 1b9387bd72f5a8e9bf8274f5764b0db0176fdba2
extra : source : 0cab9a382d817e6fbab9daa37db0f23e7f73e71f
The file is a filtered version of the make file that we previously
started generating for client.mk. Why there is special casing for
UPLOAD_EXTRA_FILES, I'm not sure. This smells fishy and is something
I'd like to take a look at once all code is ported out of client.mk.
The removal of the logic from client.mk meant that we could remove
a bunch of code from client.mk related to loading mozconfig files.
We can now simply include the auto-generated make file directly and
be done with it.
MozReview-Commit-ID: 4M5NElQA7iR
--HG--
extra : rebase_source : 87ed98fa62513007c6fdd2df00eb871a5a29a146
extra : source : 247617a64b7c438528f97d10c86e2f7b8cb72237
We write the file that client.mk is printing from Python. We can
also log it from there pretty easily. So do that.
MozReview-Commit-ID: 7eeugdOJR5b
--HG--
extra : rebase_source : 308826e948fa20684bbc40c806322f802689627e
Upcoming commits need to move more logic from client.mk. It will
be easier if we have a list of lines in the mozconfig as a local
variable.
MozReview-Commit-ID: 1wFZTfWLGP9
--HG--
extra : rebase_source : 5e3c408fdf7f953e9cbac1c4a57fd5fa87f8fbbc
Compiling C/C++/asm just requires the list of sources from _process_unified_sources() and the
Sources/GeneratedSources objects, along with the ComputedFlags and
PerSourceFlags.
The assembler invocation will need to be tweaked to support yasm as
well, but this works so far for the xpcom directory.
MozReview-Commit-ID: 91BSKV9XPLU
--HG--
extra : rebase_source : c71968f84f9f43b19afa3f4e354a3b30d8315295
Detecting the upstream repository isn't always possible. Using 'git merge-base'
seems to be a more reliable way of detecting outgoing files.
MozReview-Commit-ID: 3zOdTRIcggw
--HG--
extra : rebase_source : 56dfd16bbddca38195f6a833dc5696cf509f3fa8
In addition to help fix this bug, having the base revision will allow us
to download the most likely relevant artifacts from CI. For example, in
|mach try| we can search for a 'full-taskgraph.json' artifact from the
most relevant decision task, even if not updated to the latest central.
The ability to find the base revision will also be needed when merging
tools/tryselect/vcs.py into mozversioncontrol.
MozReview-Commit-ID: 2WRRs4AoXcK
--HG--
extra : rebase_source : 28d759ffaece14dd260402fa730136288ab8111b
Our toolchain detection logic checks whether we can reuse the target
C (resp. C++) compiler for the host compiler. This is generally only
applicable in the not-cross-compiling case, but we had special logic to
check for clang in the cross-compiling case and accept it, as clang is
able to generate code for multiple architectures from a single compiler
binary.
Our recent switch to clang on Android has exposed a problem in this
logic: we would never check whether the target clang, compiling for the
host, could actually find the host's headers. This was especially
problematic on OS X hosts, where the host clang contains special logic
to grovel inside the XCode installation to find C++ headers. The clang
from the NDK, however, was ignorant of the XCode installation.
Therefore, the NDK clang would happily compile code for the host, even
including C headers for the host, but would be hopelessly lost when it
came to compiling C++ headers during the actual build.
In hopes of mitigating this, we now include a check for a representative
header for C and C++ when checking compilers for each of those
languages. This check will detect such problems as the above, and will
also alert people to potentially misconfigured compilers in other
situations.
We need to modify our test framework to cope with headers being
included, since our mock environment isn't actually equipped with a full
set of compilers and headers.
b6adf66f34c6 (bug 1416052) changed the value for "fh" when this code
is called. It can now be an io.BytesIO. This type enforces that
arguments are bytes and doesn't perform automatic type coercion like
most other parts of Python 2.
self.topobjdir is a unicode. And unicode_literals isn't in effect
in this file. So convert self.topobjdir to bytes to make BytesIO
happy.
MozReview-Commit-ID: LrWTKFp3ZKT
--HG--
extra : rebase_source : dc8b9cabe199c78073c96b1c16f9960f92e399e4
Forgot to push the updated patch to review.
Test-only change. So self-reviewing.
CLOSED TREE
--HG--
extra : amend_source : 44bbf8748ce10982e6a3d96ff9ed42d3dc653f3f
The clobber logic is already written in Python. Now that we
always use mach in front of client.mk, we can check the clobber
state before we execute client.mk.
Since we always check the clobber state, we can remove the
CLOBBER files from various dependencies in client.mk. The
clobberer code should ensure everything is in a good state.
The refactor of the clobber Python code required some changes to
its testing. We drop some support for verifying output strings.
But testing this correctly would require a bit of effort. I don't
think it is worth it.
MozReview-Commit-ID: 69CoImCgtNm
--HG--
extra : rebase_source : c925bb49fd54fe6a5abaa4ac9dc0833e139c6a57
Now that mach is being used to invoke client.mk, we can perform
objdir directory creation there.
Removing the use of mkdir_deps meant that we could remove some
included make files which AFAICT were only used to provide
$(mkdir_deps).
MozReview-Commit-ID: 4ZRToz8xqZy
--HG--
extra : rebase_source : 8d0d2430b33863e1dec8cee84e72178307d1c6e0
extra : source : d223afd90123eba035714288d5da9394b2dbb8d8
This should have the same net result.
TBH, I'm not 100% convinced we need this export. It is only needed
to send variables to sub-makes. And the only make file reading
FOUND_MOZCONFIG is client.mk. Since the code that evals the
auto-generated make file is always executed in client.mk, we
shouldn't need this export.
All this code is going away soon anyway. So I'm inclined to cargo
cult this just in case.
MozReview-Commit-ID: DqF1BU702A
--HG--
extra : rebase_source : 31859c0d4bb6ceb06367bf0ca554d79d57d2d0c3
Currently, client.mk calls `mach environment` to obtain a make file
to be evaluated in the context of client.mk. The reason it is
implemented this way is because client.mk could be an entrypoint to
the build system.
With recent changes that require the use of mach to use client.mk,
we are now guaranteed to have Python code running before client.mk
is invoked. This means we don't need to invoke `mach` from client.mk.
This commit ports the code for generating a client.mk suitable make
file from `mach environment` to the build dispatcher. We now write out
a new .mozconfig-client-mk file in the objdir. client.mk is changed
to cat this file and to include it as a native make file.
The OBJDIR environment variable is also set so client.mk knows where
to read the auto-generated file from.
This commit should be backwards compatible.
Hopefully it is obvious, but this new make file is only temporary.
As soon as the remaining mozconfig logic is moved out of client.mk,
we should be able to simplify down to a single "include" in client.mk.
MozReview-Commit-ID: BEfWo76Z1qA
--HG--
extra : rebase_source : 752df93f816b95bda108a3c787d7f4941b136bbb
Actual uses of this variable were removed by 5fb427c50ca3
(bug 1412431). This is dead code.
MozReview-Commit-ID: GsDqpXZol2L
--HG--
extra : rebase_source : ce6eb799aa8a20e7a9af0587167bf05a6095b8df
Instead of evaluating config.guess in client.mk, we evaluate it
in Python. The Python code also looks for CONFIG_GUESS in the
mozconfig. This still happens in client.mk courtesy of evaling
the mozconfig's relevant parts.
MozReview-Commit-ID: 87NmQiB2ccX
--HG--
extra : rebase_source : 368bc7bf1375a3943ce62fbb77458c40091a7092
We're about to start moving logic from client.mk into Python.
The transition will require doing things like setting
environment variables when we invoke client.mk. Prepare for
this by factoring the "call client.mk" logic into a reusable
function.
The function will obviously be temporary until client.mk is
no more. I'm not sure what the end state will be yet. But surely
having equivalent code in Python is better than having it in
make.
MozReview-Commit-ID: F15DxXSV8V3
--HG--
extra : rebase_source : e04815201652cd90a6e5e1b105be761c21169aa6
This is fairly straightforward to represent as a GENERATED_FILES, though
we have to take some care to construct the outputs tuple correctly. This
script needs to run during export, and unfortunately none of the STL
headers have proper file extensions, so the 'new' header is
special-cased in the recursive make backend to serve as a marker for
running it in the correct tier.
We can't remove the stl-headers file yet because it is still used for
the system header generation.
MozReview-Commit-ID: 3tQTOY0LAsQ
--HG--
extra : source : 828d43ec1b16edaac69c42f15561f26e209051f1
Before, I/O errors writing to stdout/stderr (e.g. due to broken pipe)
would result in handleError() being called and execution would
keep running. This would potentially result in an error message for
every log/line failure being printed to stderr.
We change the behavior so I/O failures are fatal and abort
execution.
We test the new behavior by changing a test to pipe to `head`
directly. Since `head` exits once it has seen sufficient output,
this results in an EPIPE which now results in immediate program
termination.
MozReview-Commit-ID: 1UecZJ56h4r
--HG--
extra : rebase_source : b35d9c096621d9698260d29a7d0132c4989200a7
This makes `configure --help` work again (it stopped working due to
370a4a2a7898 (bug 1405982).
MozReview-Commit-ID: 2gCmAAsE0qW
--HG--
extra : rebase_source : 5b61b4972c514bb69b239777e0b354ac0357291e
This doesn't need to run when the context manager is active.
MozReview-Commit-ID: DV3SZyUwxlM
--HG--
extra : rebase_source : b32b081cbe9298fb46cbccdf511f9b89b92cf643
We currently print compiler warnings summary then stop the build
output manager, which basically stops resource monitoring.
A future commit will refactor this code a bit and it will be easier
to have the monitor.stop() call directly after the build finishes.
MozReview-Commit-ID: 3uWFWG6ELnH
--HG--
extra : rebase_source : 0d006c99d2ee0ccda49bfa81c31ccb47541a23ce
While we're here, also move the low-level code to invoke configure
to a proper Python module.
MozReview-Commit-ID: 4rlCxOwcVu1
--HG--
extra : rebase_source : ec16ee949743c3a43940d46b44c43a2895bfa250
extra : source : b4f9bef50278db75b1b6229635c877e07dffb1bc
The goal is to move this lower-level code somewhere where it can be
reused.
The only meaningful code change as part of the move was handling
of the mach context. Before, we could access self._mach_context.
After, we need to pass it as an argument. This also uncovered a bug
where "telemetry_handler" could be None but we called it
unconditionally. We now guard this call with an "if."
MozReview-Commit-ID: FWw4l6H1ayS
--HG--
extra : rebase_source : 97fb0db8d6fea32435f8da999e34a8e023f98339
extra : intermediate-source : bf7cfe592722018dd0894002e41f4997b36cd11b
extra : source : 2f62818d80a3e31b47194d8ecb9169bab930ed7e
In general, we shouldn't have so much business logic in mach_commands.py
files.
I'd like to move more of the low-level "perform a build" logic out of
mach_commands.py to facilitate some future work. That code makes use of
this output processing code. So as the first step in moving the build
code, we move its dependencies.
As part of this, I also cleaned up some random unused imports around
lines that were touched.
No meaningful code changes were performed as part of the code move.
MozReview-Commit-ID: 96mGWUJ7oLb
--HG--
extra : rebase_source : 1cfe6e4d12bd84fecf737b9b0cdeeac16f5d51a7
Currently, marking a logger as a structured logger will require a
subsequent function call in order for the logger to be hooked up
to active handlers. This behavior is not intuitive and makes it
easy to not have handlers for newly-registered loggers. This means
messages may not be logged anywhere.
In addition, we have to manually specify which named loggers to
enable structured logging for. This can be annoying.
We change the behavior of register_structured_logger() to
automatically add existing terminal and json handlers to the
logger being marked as structured.
We also introduce an API to enable structured logging for all
loggers. Existing consumers of registered_structured_logger()
in mozbuild have been updated to use this API. A new consumer
has been added for the `mach configure` command because it should
have been there before.
We stop short of making enable_all_structured_loggers() the default.
This is because various commands interact with the log manager in
ways that will result in duplicate logging of messages and
dropping of structured messages. There is a bit of a rabbit hole
here and addressing it can be done as a follow-up.
MozReview-Commit-ID: 1aU6eJvTSMP
--HG--
extra : rebase_source : 2a0a569b378cc3083b55fc7076b291abdfb7453f
We don't use this attribute outside of this function. Besides, you can
easily obtain a handle on the logger by calling
``logging.getLogger('mach')``.
MozReview-Commit-ID: 41vdn6McowW
--HG--
extra : rebase_source : dca383e5f5b770fa3adb09d753897633258302a1
This is fairly straightforward to represent as a GENERATED_FILES, though
we have to take some care to construct the outputs tuple correctly. This
script needs to run during export, and unfortunately none of the STL
headers have proper file extensions, so the 'new' header is
special-cased in the recursive make backend to serve as a marker for
running it in the correct tier.
We can't remove the stl-headers file yet because it is still used for
the system header generation.
MozReview-Commit-ID: 3tQTOY0LAsQ
--HG--
extra : rebase_source : bd9f00e45a7bce4daaa0e1c16e22b28536658e37
Brindusan Cristian
Ted Mielczarek
Sylvestre Ledru
Axel Hecht
Gregory Szorc
Sebastian Hengst
Nathan Froyd
Csoregi Natalia
Mike Shal
Andrew Halberstadt
Masatoshi Kimura
Chris Manchester
Andreea Pavel
Tom Prince
Dustin J. Mitchell
Phil Ringnalda