Before, virtualenv.py may have attempted to use 3rd party
(untrusted) pip indices when installing wheels for pip,
setuptools, and wheel. These dependencies are vendored in
the tree for a reason. So don't let virtualenv contact the
outside world.
MozReview-Commit-ID: 6BCU0WegJO1
--HG--
extra : rebase_source : 6231235486e8cfa784e67e8b18004dd63d8aaf36
While we're addressing virtualenv foo, let's ensure we are running
the latest version. This also pulls in newer versions of pip (8.1.1),
setuptools (20.3), and wheel (0.29.0).
MozReview-Commit-ID: G5uSy66Kd6u
--HG--
extra : rebase_source : 804f230adcf77335c79a93537d9623ac3836d9bf
@depends functions are declared like the following:
@depends('--option', other_function, '--other-option', ...)
def ...
To simplify some of the processing related to those arguments it's
passed, keep a tuple of Option and DummyFunction objects corresponding
to those arguments.
For the same reasons as set_config is being moved to the global scope,
we're moving set_define to the global scope here. An additional change
is that set_define is now part of the sandbox itself instead of being
defined within the sandbox, which makes it share the implementation
details with set_config.
The way set_config is set currently makes it difficult to introspect
moz.configure files to know what configuration items are being set,
because they're hidden in the control flow of functions.
This makes some of the moz.configure more convoluted, but this is why
there are templates, and we can improve the recurring cases afterwards.
Currently, ConfigureSandbox._db stores two different kind of
information. This split those in two different instance variables
instead, making things clearer.
The way functions are being sandboxed in moz.configure land is that
their global namespace is being replaced with a limited and identifiable
dict. And we avoid re-wrapping a function that already received this
treatment.
The problem is that template functions have their global namespace
replaced, and any function that is defined within the template inherits
that global namespace. So when it comes time to wrap those functions
defined in templates with e.g. depends, we detect that they're already
wrapped although they are not, because we look if their global namespace
is of the recognizable type we use when replacing it.
So instead of looking at the global namespace type, keep track of all
functions that are wrapped.
Currently, if a @depends function doesn't have a return statement or
return None, a result is automatically set from all the set_config()
called from the function.
As we're going to move set_config to the global namespace, and as this
feature is only used once, and it's only used for something that was
written before ReadOnlyNamespace was exposed to the sandbox, we can
"safely" get rid of it.
This will be needed to teach artifact builds to extract files from
omni.ja files whose content is loaded into memory (from a tar
archive).
MozReview-Commit-ID: LH2HkKx5Zj3
--HG--
extra : rebase_source : 7f6b176f0ef9fc87889151d1d02da62de8a455d8
extra : source : 266928b5a7615fa054c70adf0f649cbb3f085e8d
This change adds a `Version` type to moz.configure which is a small
wrapper around `distutils.version.Version`. It's suitable for wrapping
version numbers in configure checks and doing equality or greater-than
less-than comparisons in a sensible way.
MozReview-Commit-ID: BOL6yvemulG
--HG--
extra : rebase_source : 3b463eac0499086f8acffda0d01418b6ab17f3d6
extra : amend_source : aebd6e40c408d9f868623b2f53fcdf7455e2fff5
In Python parens around an expression without a trailing comma is not a tuple,
so ('foo') == 'foo'. This is really easy to screw up with check_progs, which
coerces progs to a list and would give you ['f','o','o'] in this case. This
patch enforces that the progs argument is a tuple or list and errors if it
is not.
MozReview-Commit-ID: 7BJZuF9B8D5
--HG--
extra : rebase_source : 5db9a6b4bb9ef7195c2513407810093bff5e9174
extra : amend_source : f67ab46c2ac00a2a95cfc67e9763ac12b690ac14
If we add an ObjDirPath as an input to a GENERATED_FILES script, we
should run it in the misc tier to ensure the dependent files are created
beforehand. We need to use Path objects instead of raw filenames in the
GeneratedFile object so the recursive make backend can distinguish
between source and objdir files.
MozReview-Commit-ID: 9thHTi75zdI
While rare, this is something we support for e.g. --enable-eme, where it
can be either given with no values, "adobe", some future other EME GMP
adapter names, or a combination of them.
This will make it easier for binaries only archive generation to consult
the list of binaries that are relevant to packaging.
This does overlap somewhat with compile databases. Perhaps someday the
logic could converge.
While I was here, I cleaned up writing of the all-tests.json file to
avoid an intermediate string variable.
This patch adds to_dict methods on some frontend data types. There is
room to improve the serialization of these types. For example, we could
use __slots__ to drive the default formatter. We could also leverage a
custom JSONEncoder class that knows how to call to_dict() on instances.
In the spirit of perfect is the enemy of done, I think this should be
deferred to a follow-up bug.
MozReview-Commit-ID: XVnKB1MNqu
--HG--
extra : source : 4167dfdf10457360c9c94ee6e55b03ef1b92c16d
extra : histedit_source : 2f3f10bbb8ef9da8c57b0d85cbf4ea80242eff1a%2C1a5654253181d75eecd5dfaca2c87e353f8126cd
Now when we print instances in a debugger, we'll see something like:
<StaticLibrary: other-licenses/snappy/libother-licenses_snappy.a>
<StaticLibrary: toolkit/library/StaticXULComponentsEnd/libStaticXULComponentsEnd.a>
<SharedLibrary: toolkit/library/XUL>
<StaticLibrary: toolkit/library/libxul_s.a>
<HostProgram: config/nsinstall_real>
<Program: memory/replace/logalloc/replay/logalloc-replay>
<SimpleProgram: mfbt/tests/TestArrayUtils>
... instead of the the class name and memory address.
MozReview-Commit-ID: 8zdrM6KfP8U
--HG--
extra : source : 84849ad026c9ba1bbf71c93172b0a03440e51bec
This will make it easier for binaries only archive generation to consult
the list of binaries that are relevant to packaging.
This does overlap somewhat with compile databases. Perhaps someday the
logic could converge.
While I was here, I cleaned up writing of the all-tests.json file to
avoid an intermediate string variable.
This patch adds to_dict methods on some frontend data types. There is
room to improve the serialization of these types. For example, we could
use __slots__ to drive the default formatter. We could also leverage a
custom JSONEncoder class that knows how to call to_dict() on instances.
In the spirit of perfect is the enemy of done, I think this should be
deferred to a follow-up bug.
MozReview-Commit-ID: XVnKB1MNqu
--HG--
extra : rebase_source : 376c93467dde3450c7c21cff918cb34913a3c5fe
Now when we print instances in a debugger, we'll see something like:
<StaticLibrary: other-licenses/snappy/libother-licenses_snappy.a>
<StaticLibrary: toolkit/library/StaticXULComponentsEnd/libStaticXULComponentsEnd.a>
<SharedLibrary: toolkit/library/XUL>
<StaticLibrary: toolkit/library/libxul_s.a>
<HostProgram: config/nsinstall_real>
<Program: memory/replace/logalloc/replay/logalloc-replay>
<SimpleProgram: mfbt/tests/TestArrayUtils>
... instead of the the class name and memory address.
MozReview-Commit-ID: 8zdrM6KfP8U
--HG--
extra : rebase_source : 7f964ad44a0061674f77d5716a6769a4aedb9e6c
I'm not sure when this changed, but at least Visual Studio 2015
doesn't always emit a space between the line number and the ": warning"
text in cl.exe output.
Making the space optional in the regular expression enables one a
VS2015 build to capture 375 warnings instead of 17. We still fail to
capture some warnings (notably generic warnings about bad command
arguments and linker warnings). But that can be dealt with later.
MozReview-Commit-ID: q402CxTrQK
--HG--
extra : rebase_source : 6376a65b8d8145d68bad9b795b6abd6f4becefbd
Virtualenv will sometimes find a different executable from its sys.executable on OS X,
causing a check in the build system comparing filesizes between sys.executable and virtualenv
python to fail, resulting in clobbering and re-building the virtualenv every time the virtualenv
is activated, causing the build backend and more to be re-built.
Instead of checking file sizes directly, this commit causes us to record the size and version of the
Python executable that created the virtualenv. If the Python executable checked is not the virtualenv
Python, or we have a different version than was used to create the virtualenv, then the virtualenv is
considered to be out of date.
MozReview-Commit-ID: KmrVfQCtbS3
--HG--
extra : rebase_source : d2b87325e10da6dfcd74f8b0d2ef7c0efb71595b
Virtualenv will sometimes find a different executable from its sys.executable on OS X,
causing a check in the build system comparing filesizes between sys.executable and virtualenv
python to fail, resulting in clobbering and re-building the virtualenv every time the virtualenv
is activated, causing the build backend and more to be re-built.
Instead of checking file sizes directly, this commit causes us to record the size and path to the Python
executable that created the virtualenv. If the Python executable checked is not the virtualenv
Python, or the python that was used to create the interpreter, then the virtualenv is considered
to be out of date.
MozReview-Commit-ID: KmrVfQCtbS3
When writing e.g. Japanese text files on Windows, the text is encoded in
mbcs. So if a mozconfig is edited this way, and contains non ascii
characters, they won't be utf-8 as we're currently trying to decode
them, but mbcs.
This is currently preventing a command from having both args and subcommands at the same
time.
MozReview-Commit-ID: 66frAqamGjv
--HG--
extra : rebase_source : 3bbd1cb508e5aab7ba7f9936c0c486ed8f626fe9
This adds support for an SDK_FILES variable in moz.build, which creates
a FinalTargetFiles object to install files into dist/sdk/
MozReview-Commit-ID: 97a5NdbZmmD
--HG--
extra : rebase_source : ad8d521ec56fe4610437c8d2d503c545894b40c2
The changes made to PATH by activating the in-tree virtualenv cause some git-cinnabar commands
in subprocesses to fail, because git cinnabar is expecting mercurial to be installed, while it
is not available from the in-tree virtualenv. To work around this, the changes to PATH made by
virtualenv are undone, allowing git-cinnabar to find system python, which is likely to have
mercurial installed.
MozReview-Commit-ID: 47ceHvMmuyf
--HG--
extra : rebase_source : 199d3b01fc49b2760ee14a08d4400a1addf83eac
On a CLOSED TREE because DONTBUILD NPOTB
MozReview-Commit-ID: 56vyz2CRJsU
--HG--
extra : amend_source : 4ec60bc95019147225479c32b6982dc33c649cc4
extra : histedit_source : c3dc78da75a8f5b3985024a7d73ac92ab80628c2
This moves all the reading mozconfig, finding autoconf, refreshing the
old configure, and running the old configure into sandboxed
moz.configure. This effectively bootstraps the sandboxed python configure.
Generally speaking, the configuration needs forward-slashes in paths.
We might as well make it hard(er) to set configuration items with
backslash separators on Windows by exposing mozpath.* functions in place
of os.path functions. The downside is that functions explicitly
importing os will still get the real os.path functions.
The upcoming move of the configure.py initialization to sandboxed
moz.configure changes the path separators for topsrcdir and topobjdir
from native to always use forward-slashes, which confuses the hell out
of the test_base.py test. Settle the issue by declaring that
MozbuildObject will always use forward-slashed paths for topsrcdir
and topobjdir.
The nice side effect is that now we can have actual dicts for defines
and substs from the start, which simplifies so things, although it
requires adjustments to some unit tests.
Creating a .metadata_never_index file in a directory apparently
disables Spotlight indexing.
MozReview-Commit-ID: Di4DGZK6VOL
--HG--
extra : rebase_source : feffa0785174b11cf88a2b7ffdbd9aa5eeb98ac7
extra : amend_source : 838aca018ab91720b6b29b64b846a3ef107b6c88
Using ccache can have a big impact on compile times but we don't currently
capture ccache statistics.
MozReview-Commit-ID: CdrScyAh64I
--HG--
extra : rebase_source : 70dca7ae153ae1c7ce3a241820ab7d18cb6aaa0d
We need to consider CONFIGURE_SUBST_FILES as generated files when
processing FinalTargetFiles and related variables. Additionally, we have
to make sure that we actually recurse into such directories during the
export phase.
MozReview-Commit-ID: 6ZwHMzjoT6t
--HG--
extra : rebase_source : e39e26ad8b84bcaa757c56048c9c80f5be270d9c
The Windows content indexing service has been known to scan the objdir.
This can add significant system overhead and slow down builds or
subsequent operations. The objdir is meant to be a short-lived black box
and there really isn't a major benefit to indexing it.
There is a file attribute on Windows that disables content indexing.
This commit adds a utility function for creating a directory and
optionally disabling indexing on it. We call it at the top of
`mach build` to ensure the objdir as content indexing disabled.
MozReview-Commit-ID: 68gxCxbkVAN
--HG--
extra : rebase_source : 8e95d9b2923dccadbe54288bea25883937e2f004
We don't process IPDL and WebIDL files during artifact builds. The
backend shouldn't need to care about them.
Before: 2001 total backend files; 2001 created; 0 updated; 0 unchanged; 0 deleted
After: 1945 total backend files; 1945 created; 0 updated; 0 unchanged; 0 deleted
After this change, we no longer write any .cpp files to the objdir
during config.status for artifact builds.
As part of this, we stopped generated webidlsrcs.mk and the build broke
because of an unguarded use in a Makefile.
After this change, only 102/3366 files in the objdir after
`mach configure` are not a) in _virtualenv b) named backend.mk
c) named Makefile (~950 each of backend.mk and Makefile).
MozReview-Commit-ID: 11AIn1i4x4f
--HG--
extra : rebase_source : ac90bb9f6be8b7b986aa0a61b3ccc203a18346a6
I can't think of a good reason why unified C++ files need to exist
during artifact builds. Let's not write them in this build config.
Before: 2517 total backend files; 2517 created; 0 updated; 0 unchanged; 0 deleted
After: 2001 total backend files; 2001 created; 0 updated; 0 unchanged; 0 deleted
No measureable change in performance on Linux. But on Windows where file
creation is in the ~1ms range, this will surely result in a speedup of
several hundred milliseconds.
We are still generating some .cpp files during artifact builds. This will be
addressed in subsequent commits.
MozReview-Commit-ID: Lqx36YE8qZZ
--HG--
extra : rebase_source : 8af5a9915e78af7aee2aa335552689fe33975400
Future commits will add a number of consumers. This will cut down on
boilerplate.
MozReview-Commit-ID: 8h4VWBXXd8O
--HG--
extra : rebase_source : 3376f015f1766eedcef60e9393b6d68474ffb634
As previous measurements have shown, creating/appending files
on Windows/NTFS is slow because the CloseHandle() Win32 API takes
1-3ms to complete. This is apparently due to a fundamental issue
with NTFS extents. A way to work around this slowness is to use
multiple threads for I/O so file closing doesn't block execution
as much.
This commit updates the file copier to use a thread pool of 4
threads when processing file copies. Additional threads appear
to have diminishing returns.
On my i7-6700K, this reduces the time for processing the tests install
manifest (24,572 files) on Windows from ~22.0s to ~12.5s in the best
case.
Using the thread pool globally resulted in a performance regression
on Linux. Given the performance sensitivity of manifest copying,
I thought it best to implement a slightly redundant non-Windows
branch to preserve performance. For the record, that same machine
running Linux is capable of processing nearly the same install
manifest (24,616 files) in ~2.2s in the best case.
MozReview-Commit-ID: B9LbKaOoO1u
--HG--
extra : rebase_source : e9fee3861a70e1da9d18448f8435f4bd3e28c647
This adds a simple schema for build telemetry data. We can make it more
restrictive once we have a better feeling for what kind of data we want
to submit.
This also moves more common data about the system to the telemetry handler.
We leave psutil derivied information in the resource usage data as not every
system will have psutil installed.
MozReview-Commit-ID: CFRq1Ow6AOf
--HG--
extra : rebase_source : 3022d8f5d20e3d4f9dc871cf2217a6dad2f22e05
self._pushhead_cache no longer exists. But self._tree_cache does!
This was causing AttributeError when running `mach artifact clear-cache`
and other misc `mach artifact` sub-commands.
MozReview-Commit-ID: CP8NL6eCfhD
--HG--
extra : rebase_source : 0afd11722e304c8e0ecd9a305023d43dff79dddd
extra : amend_source : ad3df6d780e7b968573588e9a1029f1a1c9d18b0
Currently, config.status runs `mach artifact install`. mach commands prefix
output lines with elapsed time by default. When running from `mach build`,
there will be 2 sets of times in `mach artifact install` output lines.
When config.status is run directly, there will be no times printed
except for `mach artifact install`. It is weird both ways.
Fix it by not printing lines when running `mach artifact install` from
config.status.
MozReview-Commit-ID: GVinyI4Z0qr
--HG--
extra : rebase_source : 80aa5714a0249d9974becee183e7cfde7143f556
extra : amend_source : a89bca7af847f73efd18fb0a09bc9e76d8943577
Previously, we We were running version 2.5.1 of requests. Newer
versions have several bug fixes and even address a CVE.
Source was obtained from
https://pypi.python.org/packages/source/r/requests/requests-2.9.1.tar.gz
and checked in without modification. This should be a rubber stamp
review.
MozReview-Commit-ID: 9tFSVJFfwGh
--HG--
extra : rebase_source : 985a63b3e09d7abd5089bdb29ebef8d7314c59f2
Opt-in by adding --enable-gradle-mobile-android-builds.
Gradle dependencies (including the Android-Gradle plugin) are assumed
to be present. Local developers will fetch them from the jcentral
repository.
Android-specific Maven dependencies are shipped as "extras" with the
Android SDK, and should be found automatically by the Android-Gradle
plugin.
MozReview-Commit-ID: 966XgddWgEu
--HG--
extra : rebase_source : 8e8c6156e1d06813c250662e104fd14c621d91ab
extra : source : 306cf0271d3e3a344fcbfd2baf75e0450c288cf1
extra : histedit_source : d17446714236c408699a0953882e84ac3a192380%2Cc21b166af79ef1e00215748820bc2670405ac1dc
The moz.build data is now sufficient to, with some convolution, generate
the same compilation database that recursing the tree with the showbuild
target does.
The resulting code is not the prettiest, and exposes the shortcomings of
the current moz.build data model. It is however a first step towards
fixing those shortcomings, because they are now more clearly identified.
This was validated on all platforms on try by checking the output of
mach build-backend -b CompileDB -d -n
is empty when backing out the patch after running
mach build-backend -b CompileDB
once.
There are a few difficult directories to handle, with limited usefulness
compared to having the CompileDB properly filled for everything else
in a timely manner, so skip them for now.
The most common issue I'm hearing with eslint is people who have an outdated
node installed. This does a quick check to verify the version is high enough
before linting.
MozReview-Commit-ID: Em0jn18OUYo
--HG--
extra : rebase_source : 5325eb5f556f93e09d48fb123e0abb625aa77b84
This adds a substs field and cherrypicks the MOZ_ARTIFACT_BUILDS field so
we can determine whether or not an artifact build occurred.
MozReview-Commit-ID: 8aio8mP8pmR
--HG--
extra : rebase_source : aa0dc2b7ef61e8b02d202c3a631d276e019d3dfd
This adds a substs field and cherrypicks the MOZ_ARTIFACT_BUILDS field so
we can determine whether or not an artifact build occurred.
--HG--
extra : rebase_source : 8878751b0bf5159cb8baedc68c9ab88cf8563628
This adds a post_dispatch_handler hook that will be called after each
mach invocation and uses it to submit data to telemetry.
--HG--
extra : rebase_source : 1dedea568b7ccec17fa58eb1841425b165d8a644
Because the add-ons manager hasn't startup up yet we can replace the certificate
database in xpcshell tests with one that claims add-ons are signed by valid
certificates even when they aren't. This allows us to run tests even in builds
where signing cannot be disabled during for the normal application.
This adds an override for all tests except those that are explicitely testing
signing.
--HG--
extra : commitid : 24s3ni5gVYe
extra : rebase_source : a95571dc3556bb035511eea424ba57e8c7007eba
Because the add-ons manager hasn't startup up yet we can replace the certificate
database in xpcshell tests with one that claims add-ons are signed by valid
certificates even when they aren't. This allows us to run tests even in builds
where signing cannot be disabled during for the normal application.
This adds an override for all tests except those that are explicitely testing
signing.
--HG--
extra : commitid : 9Zd5aH92lAQ
extra : rebase_source : 880d0fff51205dc92df91a61fef8b246d2ea2123
This fails under Docker otherwise. platform.architecture() uses the
architecture of the Python executable, which should be fine assuming the
system-installed Python is being used. Even if the user installed their
own Python, running a 32-bit Python on a 64-bit system feels like
something that would be extremely rare.
DONTBUILD (NPOTB)
--HG--
extra : commitid : HowMKbWsoTt
extra : amend_source : 9c6fd75da6d521845d561e04350b2e82dccf2b38
We need this package to build psutil and other Python packages with C
extensions.
Fedora 23 offers Python 3 as the default package and the package name
changed from python-devel to python2-devel. Fedora 22 does appear to
support python2-devel as an alias. So we use the same name everywhere.
DONTBUILD (NPOTB)
--HG--
extra : histedit_source : 7f0f9930c84f1cff396595309d47e1d600ed2609
extra : rebase_source : 544bcaa84b52ed036e76ba2a44a6074c761790cc
extra : commitid : IgEon98g61O
extra : source : 515c94229a9150246dc88318e92216d3a6d68a39
extra : amend_source : 88f419a5374ff7324dbf5f9a4a33ed313e2e2470
"git" is ambiguous between dev-ruby/git and dev-vcs/git.
--HG--
extra : histedit_source : 4d38547125f11114ee6f5a20f859c475aaa942a9
extra : rebase_source : 9ded586dd5e79953e15849e3aa27c57084fb4de0
extra : commitid : KDg9b0sFs5p
extra : source : d2cfa7eb10fc6139091de6530fa92e763eafd077
extra : amend_source : c8318134a0c8dc16d339e4943f737392e917c4eb
This will perform a single HTTP request that completes in <1s. Contrast
with before when we performed multiple HTTP requests and the process
took several seconds.
DONTBUILD (NPOTB)
MozReview-Commit-ID: DjX3LBdSOIk
--HG--
extra : commitid : Li155pLmmc
extra : rebase_source : b098d506fbfaab2b4a7946af48fd49700b1845dc
extra : amend_source : b4142e5de891160593853ea79dd14ea1d64f03ba
We have very few directories where we have SOURCES declared that are not
part of a library or program in some way. In fact, there is only one
where it is legitimate because we only use the object file
(build/unix/elfhack/inject). Others are the result of moz.build control
flow (see e.g. netwerk/standalone), and we end up building more objects
than we need to.
There are other cases where we need objects without actually linking
them anywhere, but there are other sources in the same directory, and a
corresponding Linkable is emitted. And in fact, the only case I knew
about (media/libvpx), doesn't use such objects since bug 1151175.
Since bug 1239217, artifacts builds are using a hybrid build system that
uses the fastermake rules to copy files to dist/bin, which means
artifact files are not removed by the processing of the dist/bin install
manifest. This means we don't need to add them to the recursivemake
install manifest anymore.
Currenlty --enable-artifact builds will take artifacts from fx-team regardless of the
state of the current working directory. This can lead to broken builds if someone
updates to a tree other than fx-team.
This commit changes the default behavior from tracking fx-team to finding all recent
pushheads and finding the closest to the working parent that has artifacts available.
This also fixes a mis-match between tree names according to mozext and branch names in
the taskcluster index preventing artifact download from common integration branches.
--HG--
extra : commitid : 3LqLna4qxk0
clang based tools typically choke when they encounter a lot of -XClang
arguments, and since those args are passed to the cc1 driver they're
unnecessary for such tools anyways.
This helps people who have things like --enable-clang-plugin in their
mozconfig use clang based tools without having to remove these arguments
manually.
Downloaded from https://pypi.python.org/packages/source/v/virtualenv/virtualenv-14.0.1.tar.gz
and extracted without any modifications.
This appears to speed up `mach configure` for artifact builds by ~3.3s on
my Linux machine (~16.3s to 13.0s). A significant part of this appears
to be newer version of pip caching and reusing wheels after first
install.
This should be a rubber stamp review, as all changes are from trusted
upstream packages.
--HG--
extra : rebase_source : a38af27ab45f21e9ea943fcd23655b3ce51ffdd8
Calling error() in moz.build files to indicate unsupported configurations is
useful, but readers using EmptyConfig will trigger them currently. This patch
allows a Config to have an `error_is_fatal` attribute, which will make
error emit a warning instead.
--HG--
extra : commitid : 8rqc0rx4iOK
extra : rebase_source : 0bece15b1d0fcbb38e01a26101c1f4697285c583
This is two straightforward optimizations in FileCopier: avoiding a redundant iteration
over the directory structure to find destination files (which includes a
call to normpath) and avoiding redundant calls to determine directories to preserve
when remove_unaccounted is not specified (which include a call to dirname).
Running a no-op install of _tests with this patch results in a reduction of about
25,000 calls to normpath and remove about 220,000 calls to dirname, resulting in
an overall speedup of 10-20%.
--HG--
extra : commitid : 8nyTo489q8X
Listing the tests in all-tests.json is prerequisite to adding support
for Marionette tests to `./mach test FILE`.
Marionette tests will be run from the source directory, as they do not
currently need packaging.
--HG--
extra : rebase_source : 41b80531b671cb2f4cca4337c41ba94ab189f8c0
The current rule is only for "backend.RecursiveMakeBackend", but, with
the current default of generating both the RecursiveMake and FasterMake
backends, the command creates/refreshes both backends. This is, in fact,
how the FasterMake backend is refreshed in most cases.
Moreover, with an hybrid backends, the generated file is not
"backend.RecursiveMakeBackend" anymore, so we need a more generic way to
handle this.
Furthermore, it's not necessarily desirable for all backends to have a
dependency file to handle the dependencies to refresh the backend, so
generate a plain list instead. This has the side effect of making `mach
build-backend --diff` more readable for changes to that file.
Finally, make the backend.* files created like any other backend file,
such that its diff appears in the `mach build-backend --diff` output.
The FasterMake build system is meant to be invoked through `mach build
faster`, which does it already, or, in the near future, as part of an
hybrid build system, which will deal with it as well. People doing
`make -C objdir/faster` won't have the backend automatically refreshed,
but that's not a supported way to use it anyways.
Install manifests are not empty in normal conditions, apart a few
exceptions where they are only used for a "magic" `rm -rf`.
However, we're going to introduce changes that will empty some of
the install manifests and make their work happen from a different
backend, in which case we don't want them to correspond to a `rm -rf`.
When doing build system changes affecting backend-generated files, I
often use `mach build-backend --diff`. But most of the time I end up
wanting to look at the full diff again when doing further changes, which
leads me to stash my changes away, run `mach build-backend` to get the
initial state again, unstash and rerun `mach build-backend --diff`.
This has been a time drain for long enough :)
The code doing this was added before we had install manifests, back when
they were purge manifests, and before we tracked all files created by
the backend. Nowadays, if an install manifest is removed, it will be
removed in BuildBackend.consume.
In fact, purging the install manifests in the backend itself breaks the
deleted files count displayed after reticunating splines.
XPI_NAME affects no tier on its own, as it merely changes paths where
things end up that are defined by other variables.
FILES_PER_UNIFIED_FILE had an erroneous value.
Those are the worst offenders of affected_tiers. The rules to handle
them are all in directories that is not necessarily related to where
the variables are defined, each of which has a Makefile.in for those
rules, which forces export to go through them.
They are respectively using PP_TARGETS and INSTALL_TARGETS. Both affect
the misc tier since bug 1240671, per the *_TARGET value they set in the
backend.
This has the nice side effect of now skipping directories where test
harness files are handled by install manifests.
JAR_MANIFESTS affects the libs tiers through config/rules.mk rules.
While we could move the rules in the backend, they are too complex to
just do that now.
GENERATED_FILES impacts the export tier through the config/rules.mk
definitions, now moved to the backend itself, so that everything is
close to each other.
Those are non-passthru variables with the same property as
ANDROID_GENERATED_RESFILES, ANDROID_APK_NAME and ANDROID_APK_PACKAGE, in
that they don't affect tiers through the backend code itself, but from
the Makefile.in along the moz.build they are defined in.
The are passthru variables that don't actually affect any tier per the
backend itself. They do affect the `export` tier by way of the Makefile.in
along the moz.build defining them, and the existence of those
Makefile.in already guarantees those directories not to be skipped for
`export`.
This initiates a move off affected_tiers in VARIABLES definition to
explicit in-backend handling, which will hopfully make things clearer.
HAS_MISC_RULE is currently used to opt-in to the misc tier in a few
directories with a misc:: rule. It is, in fact, mostly used for custom
xpi creation, which will be separately addressed in bug 1240676, so it
will eventually go away entirely, but in the meantime, we send it as a
throwaway passthru.
Historically, all tiers were handled as opt-out (may_skip), until we
added the first opt-in tier (no_skip). It doesn't make much sense to
differentiate them anymore, so handle them all as opt-in.
When adding a backend, we currently have to add them in three different
places so that they appear in the right places.
Instead, keep the list in a single place.
Currently |./mach artifact| installs an old version of the taskcluster client,
which installs an old version of requests that's incompatible with commonly
installed python versions. This bumps to a version of taskcluster client that
accepts and requests version < 3, so we pick up the in-tree version instead of
installing 2.4.3.
--HG--
extra : rebase_source : 7b4f450c0492fd3c840e4f0a0cce8b42d120df9f
generate_symbols_file only supports the global defines, and completely
ignores DEFINES from the same moz.build the SYMBOLS_FILE is defined.
This fixes this misfeature.
A recent change regressed this behavior -- while an artifact build runs, it
doesn't load certain "about:" pages due to missing libraries in subdirectories
of dist/bin.
--HG--
extra : commitid : HOPgt9kMKoV
AFAICT, we don't actually access web-platform test files from the
objdir for anything except test packaging. And we already have
a mechanism for creating test archives from files directly in the
source directory. So, let's stop copying them to the objdir and
package them directly from the source directory!
The _tests install manifest reports the following change:
before: 41,977 files installed
after: 24,537 files installed
delta: -17,440 files
We still copy some WPT files to the objdir. We might be able to
eliminate these as well. However, since there are only ~200 files,
I'm not too concerned.
I manually compared the resulting web-platform zip archives from before
and after. No files were removed from the archive. However, the new
archive does gain several hundred empty directories with .gitkeep
files. This feels weird, but it shouldn't break anything (I would
think). I'm inclined to leave them for now. I'll file a follow-up
bug to deal with them (preferably by removing them from version
control).
--HG--
extra : rebase_source : d350f8fc223982c8a11a9bf542411e5ec5e44244
The faster make backend cannot support such things, so it's better to
avoid unsupported things to slip in because it happens that doesn't
break what automation runs.
Right now, each incremental build produces a new set of GeckoView
library files, but the previous files don't get cleaned up, and you end
up with a bunch of old libraries in your objdir. This patch cleans up
the old files before producing new ones.
See https://trac.macports.org/ticket/48807. Without it, reading
input can break terminals.
--HG--
extra : rebase_source : 7eab1e60d6a6279fdaf25a4789598cf96e2b8d6c
Also fix that the default merge dir in the mach command creates a directory
that's the merge make target, and thus keeps that make target from actually
running.
--HG--
extra : transplant_source : R%FE%7Eo%CDS%D1%B4%EA%DB%D0%19q%F92%A1V%91%10%5C
Collecting the list of object files compiled, while not ideal, will give us
some indication of how much work was involved in the build. This will help
with analyzing the data.
--HG--
extra : rebase_source : e9861ed5c0766e3ee8038dbec0b9267022c523eb
This adds metadata about the local system to resource_usage.json that will
be useful for analysis when we start collecting these results. Some of it
is redundant with data collected for individual tiers, but having it
available at a top level should make analysis easier.
--HG--
extra : rebase_source : 43b2f6335a4de9c7843f75a3461d2871a97b7752
This moves monitor start/end recording so it also occurs for "what"
builds rather than only for full builds.
--HG--
extra : rebase_source : 833c7311ca5ecfe6935e0a160da695acaecbdeeb
Pushing on a CLOSED TREE after network errors.
--HG--
extra : commitid : 40hmCLC8zZJ
extra : rebase_source : 44a623fdf7cb97bfaca76617062d57c6b50fd970
extra : amend_source : 710db8302b5bf88d6bf293d71f3cc001187b6015
The result of running this backend is a json file containing an array with 3 elements:
a map from resource:// and chrome:// url prefixes to corresponding objdir directories,
a map of overrides, and a map from objdir files to srcdir files (with a flag for whether
the file was preprocessed) for js file in FinalTargetFiles. A subsequent commit implements
the procedure to find an replace based on the url mapping, subsistute based on overrides,
and find sourcedir files based on objdir files.
--HG--
extra : commitid : Di6Ro5c9sTK
This mounts the downloaded DMG and copies a subset of the libraries
into the correct places in the processed archive. They'll be
installed, with paths, into dist/bin from there.
--HG--
extra : commitid : 9BXzXzIlh3I
extra : rebase_source : 29ddbe8348f27fff5205da6c40839a04d950fe0a
For Android, this just copies .so files from the APK assets/ and libs/
directories into the root, which ends up in dist/bin.
I include .ini files as a partial replacement for having the build ID
in the downloaded file name, and then don't put the .ini files in
dist/bin.
--HG--
extra : commitid : 9uLXEzBNBdQ
extra : rebase_source : 53ba0e8a7740c2a6eb48e2772f7e12ff9157d21a
Sadly, it's slow to extract the build ID from Mac OS X DMG artifacts.
It's better to sacrifice human-legible names in order to know the
final name for an artifact quickly.
--HG--
extra : commitid : I99dtzv1O2N
extra : rebase_source : c164be1314b3132fba2e1fc86cda65b12b4baa1f
It turns out to be much easier to hook |mach artifact install| into
config.status and |mach build| than to hook into client.mk.
The additional virtualenv package avoids an import error when running
|mach artifact install|.
--HG--
extra : commitid : EnfWU0uyRfQ
extra : rebase_source : f7d11fc4c542f9798712c013c4319d92d40c28e5
StrictVersion is strict about version strings, insisting on whatever
convention Python uses. LooseVersion is not as strict but is strict
enough for our use cases.
DONTBUILD (NPOTB)
--HG--
extra : commitid : 17lNEAJhaV0
extra : rebase_source : 0a0cefa47b4558401cb85c6e9b237c0d6cf0e7fb
extra : amend_source : c7360d1a2f934338ec04d5f384d4530e3e9ebbc5
3.5.2 is what is listed in `mach mercurial-setup`. These should match.
Add a comment to each file saying to change both.
--HG--
extra : commitid : FebjTovmqGk
extra : rebase_source : 50490c1896a4c402f27cf4154b155932614da558
extra : amend_source : 73ae0ddc9f2770351d2ee2aaf5121656fb7e5750
Limit ourselves to include paths for now, because there are tricky things
involved in making this globally.
While here, use shell_quote instead of manual quoting for those paths.
With all include flags now using absolute paths, there is no need to try
to post-process them when getting them for CompileDB and codecomplete.
As a matter of fact, doing so fixes the flags in media/gmp-clearkey/0.1,
since they use literal "-include stdio.h", which was wrongly transformed
into "-include $objdir/media/gmp-clearkey/0.1/stdio.h".
MOZ_DEBUG_DEFINES are essentially defines used everywhere. So treat them as
feeding the initial value for DEFINES in each moz.build sandbox. This allows
the kind overrides that was done in the past by resetting MOZ_DEBUG_DEFINES
in Makefiles.
Currently, one needs to define DEFFILE or LD_VERSION_SCRIPT appropriately,
and somehow deal with the fact that their input format is different, which
currently relies on manual invocations of the convert_def_file script, with
awkward aggregations.
This simplifies the problem by using a simple list of symbols, with
preprocessing, allowing #includes.
We want to move it to CommonBackend, so it's better to make it more
independent, which the Defines instances now attached to ContextDerived
instances allow.
Like with ChromeManifestEntries, reloop in consume_object, with the double
goal of allowing to reuse the jar manifest handler code in other backends
and avoid code duplication in the FasterMake backend itself when support
for e.g. GeneratedFiles is added.
Instead of filling the install manifests accordingly, reloop in
consume_object, so that the jar manifest handler code can eventually
be reused in other backends.
Currently, only css files added through jar manifests are treated this way.
There is really no reason for the discrepancy, but there are actually no css
files added directly through moz.build, so this was never a problem.
On the other hand, it makes things simpler in a world where jar manifests are
treated as if they were entirely described in moz.build (which is where the
FasterMake backend is heading).
Again, this is not strictly necessary but allows to confirm idempotence of
further changes. And it has the nice side effect of making chrome manifest
files more consistent.
Using TEST_DIRS is nothing more than a shortcut for
if CONFIG['ENABLE_TESTS']:
DIRS += [...]
As such, we might as well remove it being a separate variable, and use some
Context magic to just fill DIRS when ENABLE_TESTS is set.
The security/manager/ssl/tests/unit/moz.build change ensures that the order
of DIRS before the change is kept, not because it matters, but because it
allows to confirm that nothing else is modified by this change.
Bug 1191230 added override lines with # characters to chrome manifests
for Windows.
So far, chrome manifests were handled with buildlist.py like in the
RecursiveMake backend, fed with Make variables. Without proper quoting,
those Make variables are just truncated by Make on the first # character,
and this results in mach build faster failing because of that.
However, the reason why chrome manifests were handled with buildlist.py
originally is that not all chrome manifest entries were known to the
FasterMake backend, but they now all are.
So instead of relying on Make variables and buildlist.py, we can now
rely on the newly added install manifests feature allowing to create files
with a given content.
From the backend perspective, CONFIGURE_DEFINE_FILES is the same as
GENERATED_FILES because in both cases a GeneratedFile object is emitted, but
from the perspective of some checks in the emitter, they aren't the same,
and that causes errors when adding a CONFIGURE_DEFINE_FILES to e.g. EXPORTS.
Also removes related unused variables in mach_commands.py.
--HG--
extra : commitid : IiDVMuEZtA5
extra : rebase_source : 575a51dd0ad5450323b4da5f441f8e5d721e41d6
From the backend perspective, CONFIGURE_DEFINE_FILES is the same as
GENERATED_FILES because in both cases a GeneratedFile object is emitted, but
from the perspective of some checks in the emitter, they aren't the same,
and that causes errors when adding a CONFIGURE_DEFINE_FILES to e.g. EXPORTS.
Running old extensions with newer versions of Mercurial may crash `hg`
due to the old extension accessing something or doing something that has
been changed in the new release.
To minimize the risk of this happening, we disable common 3rd party
extensions when cloning or pulling as part of `mach mercurial-setup`. We
don't want to disable everything because some extensions (like
remotenames) provide features the user may want enabled as part of the
clone/update. This leaves the door open for more failures. Hopefully
this approach is sufficient. We can always revisit later.
--HG--
extra : rebase_source : 92e7d8fe227f29fc64c0f69021bd731ba762faf3
In order to use StrictOrderingOnAppendListWithFlags instances in
mozbuild template functions, we need += to work correctly. This patch
implements extend and the associated functions (including +=),
disallowing some behaviour where convenient.
There's a subtle point hidden in the isinstance() tests: before this
patch, it was not easy to compare two
StrictOrderingOnAppendListWithFlags instances to see if they had the
*same* set of flags. That was because two instances may not have the
same class, and would only share the common
StrictOrderingOnAppendList, which isn't enough to infer the presence
of flags. To be slightly more clear, concrete instances will have
class StrictOrderingOnAppendListWithFlagsSpecialization (although
there are still multiple instances of that class) and all extend from
the unique class StrictOrderingOnAppendListWithFlags.
--HG--
extra : commitid : AMVDYt8khR
extra : rebase_source : 1ce0698691fc03fbdf6a976e92017c1d60bad15d
extra : histedit_source : 4812a565179fb4fac2e4b5cd89c4efe74e794dfa
DONTBUILD NPOTB
The top source directory configuration requires
mobile/android/gradle/m2repo/**, so it stays. There's no value
changing the location; it contains an Android-specific Gradle plugin.
We note the removal of |mach gradle-install| and point to the new
documentation.
--HG--
extra : commitid : 9Nhz2dnBIgY
extra : rebase_source : 32a2b8a92d57f963feac2bae28fed5a9f1b26f93
extra : amend_source : bf53a0b2d3d4ac0618bc82fe79914bdeaf1c1e0a
Both these files, are, after all, define files, like other CONFIGURE_DEFINE_FILES.
They only happen to have a special requirement for an expansion for all defines,
which doesn't need to happen through traditional preprocessing.
This change adds consistency in how configure-related headers are being handled.
This is needed to support hgwatchman.
--HG--
extra : commitid : 8D2A8YPNimB
extra : rebase_source : 7d5932aa049dfb352b93a87c2c8087dd7b324aab
extra : histedit_source : 9863189f265eca9e0b9363e13c59a7d55f5c633d
This change allows specifying objdir-relative paths in EXPORTS to enable
exporting entries from GENERATED_FILES. Objdir paths in EXPORTS that are
not in GENERATED_FILES will raise an exception.
Example:
```
EXPORTS += ['!g.h', 'f.h']
GENERATED_FILES += ['g.h']
```
Given the implementation, this should also work for FINAL_TARGET_FILES,
FINAL_TARGET_PP_FILES, and TESTING_FILES, but those are not well-tested.
This patch also renames the install manifest for '_tests' to match the
directory name for convenience in some code I refactored.
--HG--
extra : commitid : CwayzXtxv1O
extra : rebase_source : 5fb6f461fc740da9bce14bbdbfabdfe618af8803
Future improvements to process_install_manifest's --track option will require
adding data in the tracking dump that uses an install manifest form, and I don't
want e.g. switching branches or bisection to require to clobber in order to do the
right thing, so this change future-proofs the install manifest reader.
There are currently two operating modes for process_install_manifest:
- default, which removes any file in the destination directory that is not
in the install manifest
- --no-remove, which doesn't do the above.
While install manifests also have the ability to deal with files that may
be left in the destination directory some other way, that requires knowing the
list of those files in advance, which is not always possible.
For instance:
- with the FasterMake build backend, install manifests are split such that
there is one manifest per application of addon directory (to allow more
parallelism), which means there is one for dist/bin and one for several
of its sub-directories.
- With --disable-compile-environment combined with artefacts, the backends
are not aware of e.g. all the libraries and executables that end up in
dist/bin.
If we want to properly remove files when they are removed from moz.build
or jar.mn, we can't use --no-remove, but the alternative would remove those
files
So add an option that keeps a list of all the files that were installed as
part of processing the given install manifest(s). That information is simply
a dump of the install manifest, which, while it contains more information
than currently required, will allow to do smarter things in the future.
The default behavior for a FileCopier's copy is to remove all the files and
directories in the destination that aren't in its registry.
The remove_unaccounted argument can be passed as False to disable this
behavior.
This change adds another possibility, where remove_unaccounted may be a
FileRegistry, in which case only the files in that registry are removed.
This allows to e.g. only remove files that were copied from a previous
FileCopier.copy, leaving aside files that were in the destination for some
other reason.
Future improvements to process_install_manifest's --track option will require
adding data in the tracking dump that uses an install manifest form, and I don't
want e.g. switching branches or bisection to require to clobber in order to do the
right thing, so this change future-proofs the install manifest reader.
There are currently two operating modes for process_install_manifest:
- default, which removes any file in the destination directory that is not
in the install manifest
- --no-remove, which doesn't do the above.
While install manifests also have the ability to deal with files that may
be left in the destination directory some other way, that requires knowing the
list of those files in advance, which is not always possible.
For instance:
- with the FasterMake build backend, install manifests are split such that
there is one manifest per application of addon directory (to allow more
parallelism), which means there is one for dist/bin and one for several
of its sub-directories.
- With --disable-compile-environment combined with artefacts, the backends
are not aware of e.g. all the libraries and executables that end up in
dist/bin.
If we want to properly remove files when they are removed from moz.build
or jar.mn, we can't use --no-remove, but the alternative would remove those
files
So add an option that keeps a list of all the files that were installed as
part of processing the given install manifest(s). That information is simply
a dump of the install manifest, which, while it contains more information
than currently required, will allow to do smarter things in the future.
The default behavior for a FileCopier's copy is to remove all the files and
directories in the destination that aren't in its registry.
The remove_unaccounted argument can be passed as False to disable this
behavior.
This change adds another possibility, where remove_unaccounted may be a
FileRegistry, in which case only the files in that registry are removed.
This allows to e.g. only remove files that were copied from a previous
FileCopier.copy, leaving aside files that were in the destination for some
other reason.
The only use of BRANDING_FILES[...].source is in xulrunner/app/moz.build, for
the app.ico file.
This file has not been useful since the removal of the xpinstall-based
installer in bug 344236... 9 years ago.
Currently mach treats the first argument to eslint as the path and moves it to
the end of the arguments but this breaks usage like "mach eslint -f json browser".
It used to be necessary to change to the directory you wanted to lint but now
the .eslintignore is at the top level we just run from the top level. This means
the path argument doesn't need to be special anymore.
--HG--
extra : commitid : 5ozct0pVSC4
extra : rebase_source : 22132a240d8e6f4d099dbcdeb793958d7173e154
extra : amend_source : 2b9931b4283e1c84f699027e13eccc33fcdec978
The current implementation of HierarchicalStringList allows the following:
FOO.bar = [
'foo',
'bar',
]
while
FOO.bar += [
'foo',
'bar',
]
would be invalid because of the StrictOrderingOnAppendList enforcement.
It also allows to overwrite the entire list with a subsequent
FOO.bar = [
'baz',
]
while we've explicitly forbidden such things for every other list.
While in the vicinity, fix HierarchicalStringList._get_export_variable to not
call the HierarchicalStringList constructor uselessly.
This will allow a new kind of special variable where it is possible to do
FOO += ['bar']
All the current special variables are either strings (for which __setitem__ would
be called with a different string object), or a read-only dict (which doesn't
allow modifications).
Mike Hommey
Mike Shal
aleth
Gregory Szorc
Timothy Guan-tin Chien
Ted Mielczarek
Carsten "Tomcat" Book
Richard Newman
Phil Ringnalda
Chris Manchester
Ryan VanderMeulen
Steve Fink
Andrew Halberstadt
Jeff Walden
Maja Frydrychowicz
Wes Kocher
Nick Alexander
Sambuddha Basu
Myk Melez
Nicholas Nethercote
Dan Minor
Quentin Headen
Jan Beich
Patrick Brosset
Dave Townsend
Birunthan Mohanathas
Ehsan Akhgari
Syd Polk
Andreas Tolfsen
Jacek Caban
Panos Astithas
Jim Chen
Gregory Szorc
Gijs Kruitbosch
Axel Hecht
Axel Hecht
Bryce Van Dyk
Henrik Skupin
Robert Longson
Nigel Babu
Shane Caraveo
Kris Maglione
Chris Peterson
Jean-Yves Avenard
Ms2ger
Brian Murray
Rail Aliiev
Felipe Gomes
Chris Pearce
