Passing a file object intead of a string of the file's contents defers reading
the file to a place Python can free the resulting string earlier, reducing
peak memory consumption when packaging symbols and avoiding a MemoryError on
Windows 32 builders in automation.
MozReview-Commit-ID: H0R6BbjwhOu
--HG--
extra : rebase_source : 181b592ab503d0545f34d167d7c05428455b6671
We install the tooltool packages to ${HOME}/.mozbuild by default, and
${HOME} may have spaces on Windows. The arguments to the command-line
options need to be quoted appropriately.
DONTBUILD because NPOTB
We'll need this information to be able to locate mach later, since we
can't assume that mach is just several directories above the script
we're currently running.
If we're running bootstrap from inside a mozilla-central checkout, we'll
have all the necessary files that we need to install tooltool packages
via `mach artifact toolchain`. If we're running bootstrap from a
downloaded bootstrap.py, however, it's possible that we failed to clone
the repository. We need to account for that situation and provide
instructions on how to deal with it.
A consequence of these two mechanisms for bootstrap is that we'll have
to make sure we're invoking `mach artifact toolchain` correctly in both
cases, which we'll handle in subsequent commits.
Now that we're installing Stylo packages through `mach`, we need to have
`mach` available, which is only true after we've checked for a clone of
the tree. We move the block performing the installation in this commit
and then fixup resulting problems in future commits.
Using this command is more robust than our current method, and brings
several benefits, such as smart caching of the downloaded toolchain. We
change the clang package downloaded for Windows with this change, but
bindgen has been updated to work well with LLVM 5.0, so there should be
no problems.
Most complaints about compiler warnings being printed at the end of the
build (4abe611696ab - bug 1360764) center around the volume. And most
of the volume is due to 3rd party projects. Since work to suppress the
warnings in 3rd party projects has been slow, let's temporarily filter
out those warnings.
This commit adds a list of suppressed directories corresponding to
3rd party projects responsible for most compiler warnings. For
non-automation builds, we don't print the warnings. Instead, we
print a summary of how many warnings were suppressed.
Not all 3rd party projects were listed. The intent is to only list the
high volume ones so the warnings list is reasonable by default. I
believe the current implementation achieves that. But I wouldn't
be surprised if it requires refinement. Perfect is the enemy of good.
MozReview-Commit-ID: ExDRidsHROC
--HG--
extra : rebase_source : 49123193ea872a92208dc54a71b5f31208c10cd1
geckodriver is the Mozilla implementation of the WebDriver remote
control interface for Gecko, and provides an HTTPD proxy that
translates the WebDriver protocol to Marionette.
Building this as part of the Firefox build will allow us to run
WPT WebDriver tests to verify our implementation of Marionette and
geckodriver. It also makes it less painful to make changes across
projects.
This change will cause the geckodriver program to be built as part
of regular Firefox builds, except on macOS and Android, and when artifact
builds are enabled.
RUST_PROGRAMS in cross-compile environments cause the wrong linker to
be used. When this bug is fixed, we should be able to enable building
of geckodriver on macOS. This work is tracked in:
https://bugzilla.mozilla.org/show_bug.cgi?id=1329737
On Android, we may one to build a binary for the host system to use
(x86_64), instead of an ARM binary for the emulator.
MozReview-Commit-ID: FG5tmPv4iut
--HG--
extra : rebase_source : 091728fd2582458325689fc6e3d8b317428802d8
If we had an unsuccessful build, then duplicating the warnings at the
end of the compilation job would just be burying the actual error in a
blizzard of noise.
Several years ago there was a single zip file for all test files. Clients
would only extract the files they needed. Thus, zip was a reasonable
archive format because it allowed direct access to members without
having to decompress the entirety of the stream.
We have since split up that monolithic archive into separate,
domain-specific archives. e.g. 1 archive for mochitests and one
for xpcshell tests. This drastically cut down on network I/O
required on testers because they only fetched archives/data that
was relevant. It also enabled parallel generation of test archives,
we shaved dozens of seconds off builds due to compression being
a long pole.
Despite the architectural changes to test archive management, we
still used zip files. This is not ideal because we no longer access
specific files in test archives and thus don't care about single/partial
member access performance.
This commit implements support for generating tar.gz test archives.
And it switches the web-platform archive to a tar.gz file.
The performance implications for archive generation are significant:
before: 48,321,250 bytes; 6.05s
after: 31,844,267 bytes; 4.57s
The size is reduced because we have a single compression context
so data from 1 file can benefit compression in a subsequent file.
CPU usage is reduced because the compressor has to work less with
1 context than it does with N. While I didn't measure it, decompression
performance should also be improved for the same reasons. And of course
network I/O will be reduced.
mozharness consumers use a generic method for handling unarchiving.
This method automagically handles multiple file extensions. So as long
as downstream consumers aren't hard coding ".zip" this change should
"just work."
MozReview-Commit-ID: LQa5MIHLsms
--HG--
extra : rebase_source : cd029cdbbcccc1d16f03d63a5f1fdf60be5db4fd
extra : source : a0e257e346ccf3c1db332ec5903241f4eeb9a7ee
This allows us to write files coming from a finder or other source
that isn't directly the filesystem.
MozReview-Commit-ID: KhPSD0JYzsQ
--HG--
extra : rebase_source : 24db84974b54a714ba82dfad7ff68fd1a5bf656a
extra : source : ae8bce278626bc84914063f93292ac5e825eec36
Some manifest entries (e.g. skin or locale) have an attached identifier,
and there can be different entries with different identifiers for the
same chrome name. The change from bug 1366169 would consider those as
errors, while they are the expected configuration.
--HG--
extra : rebase_source : ceb08da909121a2ac0a2cdaba7970e4594dde09f
Several years ago there was a single zip file for all test files. Clients
would only extract the files they needed. Thus, zip was a reasonable
archive format because it allowed direct access to members without
having to decompress the entirety of the stream.
We have since split up that monolithic archive into separate,
domain-specific archives. e.g. 1 archive for mochitests and one
for xpcshell tests. This drastically cut down on network I/O
required on testers because they only fetched archives/data that
was relevant. It also enabled parallel generation of test archives,
we shaved dozens of seconds off builds due to compression being
a long pole.
Despite the architectural changes to test archive management, we
still used zip files. This is not ideal because we no longer access
specific files in test archives and thus don't care about single/partial
member access performance.
This commit implements support for generating tar.gz test archives.
And it switches the web-platform archive to a tar.gz file.
The performance implications for archive generation are significant:
before: 48,321,250 bytes; 6.05s
after: 31,844,267 bytes; 4.57s
The size is reduced because we have a single compression context
so data from 1 file can benefit compression in a subsequent file.
CPU usage is reduced because the compressor has to work less with
1 context than it does with N. While I didn't measure it, decompression
performance should also be improved for the same reasons. And of course
network I/O will be reduced.
mozharness consumers use a generic method for handling unarchiving.
This method automagically handles multiple file extensions. So as long
as downstream consumers aren't hard coding ".zip" this change should
"just work."
MozReview-Commit-ID: LQa5MIHLsms
--HG--
extra : rebase_source : 19ec875917546abc147b234a815e1a64c204b92a
This allows us to write files coming from a finder or other source
that isn't directly the filesystem.
MozReview-Commit-ID: KhPSD0JYzsQ
--HG--
extra : rebase_source : 10d494fc910982c3e34f4744592edca906d3a85d
We're getting an intermittent failure running `hg manifest` in CI. I
have no clue why.
What I do know is that we now have the mozversioncontrol Python package
for containing utility code for interacting with version control. It is
slightly more robust and I'm willing to support it more than I am
check_utils.py.
This commit adds a new API to our abstract repository class to obtain the
files in the working directory by querying version control.
Since I suspect cwd was coming into play in automation, I've also
added a utility function to mozversioncontrol to attempt to find
a version control checkout from the current working directory. It
simply traces ancestor paths looking for a .hg or .git directory.
Finally, I've ported all callers of the now-deleted API to the new
one. The old code had some "../.." paths in it, meaning it only
worked when cwd was just right. Since we resolve the absolute path
to the checkout and store it on the repo object, I've updated the
code so it should work no matter what cwd is as long as a repo can
be found. I'm not 100% confident I found all consumers assuming cwd.
But it's a start.
I'm not 100% confident this will fix the intermittent issues in CI. But
at least we should get a better error message and at least we'll be
running less hacky code.
MozReview-Commit-ID: AmCraHXcTEX
--HG--
extra : rebase_source : 815ae369776577ad374333920fd645d412a55148
Windows repackaging for complete mar files will also need to pull a
different value out of the application.ini file, so this code should be
shareable.
MozReview-Commit-ID: CzCoNRYcBPX
--HG--
extra : rebase_source : a5e4b31ba876d811436a7d8d15462fa85f0762f8
The chmod permissions need to be in octal format to get the expected
permissions settings. This only seems to affect the output if we run the
repackaging command on Linux, but it should still be fixed.
MozReview-Commit-ID: to4v7dkSBl
--HG--
extra : rebase_source : 9d2289511ebd05aea8a4c6d37136f258e2463dee
Python can run these files with 'python -m 7z_exe_foo', but using
'import 7z_exe_foo' is not allowed because the module begins with a
number.
See also: https://stackoverflow.com/a/17487228
MozReview-Commit-ID: 97iDdXlZJ1a
--HG--
rename : python/mozbuild/mozbuild/action/7z_exe_archive.py => python/mozbuild/mozbuild/action/exe_7z_archive.py
rename : python/mozbuild/mozbuild/action/7z_exe_extract.py => python/mozbuild/mozbuild/action/exe_7z_extract.py
extra : rebase_source : 1941986a7e6e56305b742710c73b90a3f7b5226b
It makes more sense to have the repackage commands in a separate
directory, since Windows repackaging will add several new types.
MozReview-Commit-ID: 1wA7F7k4NXf
--HG--
rename : python/mozbuild/mozbuild/repackage.py => python/mozbuild/mozbuild/repackaging/dmg.py
extra : rebase_source : b4b4f2fd5c45900aaf125928c144c15cfa1e115e
This patch stops GENERATED_FILES from being run during export for artifact
builds and prevents EXPORTS from being processed because some of these will
depend on generated headers.
MozReview-Commit-ID: BHZBFbVHwPT
--HG--
extra : rebase_source : fe21e07c5bb8ef957a0006ec6a06eb833692ebaf
As described in changeset c94e87a18096, chrome manifest entries are
reordered and don't necessarily appear in the order they have in jar.mn.
And in some cases, the order does matter: when entries with flags are
followed by entries with more broad flags or no flags at all. Nobody
should be writing entries in that order on purpose, so error out during
packaging when we detect the pattern.
--HG--
extra : rebase_source : d9617bbcbd8560503c532a13c10c8afb0fd49411
To make things simpler in configure code, as well as to allow the linter
to skip bugging about some --help dependencies, we make the following
work:
something.some_attr
where the result is equivalent to, currently:
delayed_getattr(something, 'some_attr')
Similar to how they can be combined with "|", we now allow using "&".
As for "|", it would have been better if it were "and", but it's not
possible to override "and" in python ; __and__ is for "&".
CommandLineHandler() expects argv to be like sys.argv, containing the
command name in argv[0], but various tests weren't doing that, in some
cases even leading to ignored arguments passed as argv[0].
In turn, that made lint.py only test browser/moz.configure instead of
all the project moz.configures as intended.
--HG--
extra : rebase_source : 8a87216edaa4a2fd27abb9ef74d38a254a2bbeed
The base bootstrap class has code to conditionally upgrade Mercurial
depending on its version. It is kinda broken. This commit overrides
that code in the MozillaBuild bootstrapper to always run `pip`, which
will ensure the latest stable Mercurial release is installed.
MozReview-Commit-ID: 2O1Ff7dAp4o
--HG--
extra : rebase_source : 98cba0abbebf13e8fe93673d5b46eb16e6a3c147
Previously, `mach bootstrap` would unconditionally prompt to help
configure Mercurial in most scenarios. I agree with Ehsan's observation
in a mailing list post that this behavior doesn't make sense when
running from a Git checkout, as the user probably doesn't care about
Mercurial if they are using Git.
This change doesn't completely ignore Mercurial for Git users. For
example, we still unconditionally run code that verifies that Mercurial
is installed and reasonably up to date. Changing this would be a bit
of work. But even if we wanted to change it, git-cinnabar users
would benefit from having a modern Mercurial installed. So it isn't
straightforward for Git users to ignore Mercurial completely.
MozReview-Commit-ID: 8ncHRgCsjz
--HG--
extra : rebase_source : 7945e3bf3d5283105bac517885f794fc5d7bba6d
Doing something like "not foo" when foo is a @depends function is never
going to do what the user expects, while not necessarily leading to an
error (like, when used in set_config).
It is better to have an error in those cases where it's expected not to
work, at the expense of making templates a little more verbose, rather
than silently do something the user is not expecting.
--HG--
extra : rebase_source : 87ba80eccc5606322f6dd185d5cb5fc88471e51b
Artifact builds were relying on disabling Rust source file inclusion.
The RustLibrary mozbuild class wants to know cargo's output directory
when initializing itself, but when no compile environment is available
rust.configure isn't included and the corresponding config keys
aren't available.
Skip inializing the dependent fields in that case. Since the artifact
build never tries to compile any of the rust libraries, leaving
these properties undefined works ok.
MozReview-Commit-ID: 8IzTsweSygn
--HG--
extra : rebase_source : a59fc01483fbc85766ff4445c5db7ddb1e49b87c
Chris Manchester
Nick Alexander
Sebastian Hengst
Nathan Froyd
Gregory Szorc
Ryan VanderMeulen
Justin Wood
Andreas Tolfsen
Wes Kocher
Mike Hommey
Xidorn Quan
Mike Shal
Shing Lyu
Chris AtLee
Jan Beich
Ralph Giles