On Windows in Python 2, the subprocess module requires the use of bytes with
the 'env' argument. For that reason, we would sometimes use byte strings with
'os.environ' like so:
os.environ[b"FOO"] = b"bar"
However, this is a failure with Python 3 as 'os.environ' must only be used with
the text type. This patch creates a new 'setenv' helper that ensures we create
new environment with 'bytes' on Python 2, and 'text' on Python 3.
Differential Revision: https://phabricator.services.mozilla.com/D38237
--HG--
extra : moz-landing-system : lando
It causes confusion because it's the mode used to _read_ the overwritten
file. Make that more obvious by renaming to `readmode`.
Differential Revision: https://phabricator.services.mozilla.com/D39445
--HG--
extra : moz-landing-system : lando
There is less incentive to keep things building with older versions of
clang for OSX builds, and we're going to require an objective-C feature
that was added in clang 5.
Differential Revision: https://phabricator.services.mozilla.com/D38581
--HG--
extra : moz-landing-system : lando
The preprocessed install manifest code uses a Makefile dependency file
to list inputs. If one of the inputs was missing (due to a file removal
or rename), an incremental build would fail until clobbered. We should
treat a missing input as simply being out-of-date and continue
processing.
Differential Revision: https://phabricator.services.mozilla.com/D38576
--HG--
extra : moz-landing-system : lando
Rather than relying on the mar-channel-id set in the `mar` binary, set the channel
explicitly from taskcluster. This allows us to re-use the `mar` binary between
builds/channels.
Differential Revision: https://phabricator.services.mozilla.com/D37481
--HG--
extra : moz-landing-system : lando
Credit: Callek for figuring out an issue in 'make check' making the binary
absolute in mozbuild.base.
Differential Revision: https://phabricator.services.mozilla.com/D37319
--HG--
extra : moz-landing-system : lando
When building Gecko/Android/aarch64 on Windows, `--target` parameter may not be incorrect value. Although `check_compiler`'s `info` is target compiler, clang on Windows is always detected as `clang-cl`, not `clang`.
```
c:/Users/mkato/.mozbuild/clang/bin/clang.exe -E -dM - < /dev/null
...
#define _MSC_VER 1916
```
So even if using clang on Windows, not clang-cl, we should detect as 'clang' correctly
Differential Revision: https://phabricator.services.mozilla.com/D36422
--HG--
extra : moz-landing-system : lando
By globally importing PackageFrontend from the globe-analysis module we break the logger for
the PackageFrontend package.
Differential Revision: https://phabricator.services.mozilla.com/D36887
--HG--
extra : moz-landing-system : lando
The SDK headers may not be installed in /usr/include. The usual response
has been to have people run e.g. `open
/Library/Developer/CommandLineTools/Packages/macOS_SDK_headers_for_macOS_10.14.pkg`
which is not really sustainable.
This makes builds that happen on a macOS host try to detect their SDK
and use that as a default for --with-macos-sdk, which has the side
effect of enabling the SDK version check in that configuration.
Differential Revision: https://phabricator.services.mozilla.com/D36558
--HG--
extra : moz-landing-system : lando
This test is working with minimal effort. Let's get it running to prevent
future regressions.
Differential Revision: https://phabricator.services.mozilla.com/D36098
--HG--
extra : moz-landing-system : lando
Configure should just be able to find the right one. If it doesn't, that
should be fixed in configure rather than with suggestions in bootstrap.
Differential Revision: https://phabricator.services.mozilla.com/D36562
--HG--
extra : moz-landing-system : lando
Configure should just be able to find the right one. If it doesn't, that
should be fixed in configure rather than with suggestions in bootstrap.
Differential Revision: https://phabricator.services.mozilla.com/D36562
--HG--
extra : moz-landing-system : lando
Bug 1554987 made `mach try` use a transient remote, but that causes
problems with existing setups that happen to use the same remote name,
because of a combination of not-quite-as-documented-as-it-should
behavior of git.
- `git -c foo.bar=qux` doesn't override the value of `foo.bar` from the
git configuration when `foo.bar` is an item that can take several
values.
- `remote.$remote.url` and `remote.$remote.pushurl` take several values,
allowing to give several URLs.
The combination of both means that if the git configuration already has
`remote.try.url` set, that value takes precedence (because git push
tries them one after the other, and takes the one from the command line
last)
One way we could increase the chances of things working out fine would
be to use `remote.try.pushurl`, which if already set, is more likely to
be right than an existing `remote.try.url`.
OTOH, it turns out, after more investigation, that bug 1554987 requires
a footgunny setup to happen in the first place. Namely, it requires
having run `git lfs install` from a git-cinnabar clone.
so we just go back to the previous status quo.
Differential Revision: https://phabricator.services.mozilla.com/D36149
--HG--
extra : moz-landing-system : lando
See https://stackoverflow.com/a/24026735.
Adding the `docs` package requirement is not ideal, but it's not worth
the effort to install it only in automation (or in the relevant task),
and it's not *that* large: 1.0G on my macOS installation.
Differential Revision: https://phabricator.services.mozilla.com/D35834
--HG--
extra : moz-landing-system : lando
4.3.3 is the strict minimum required for v-c-t's config wizard, but it
is preferable people use more modern versions. We could go with 5.0, but
it feels like people still using 4.8 and 4.9 don't really need to be
bugged to update to a more recent version, they are kind of modern
enough. OTOH MozillaBuild comes with 4.5.x, and this will force an
upgrade for those.
Differential Revision: https://phabricator.services.mozilla.com/D35756
--HG--
extra : moz-landing-system : lando
This makes the bootstrap behavior wrt. Mercurial consistent on all
platforms, making Windows bootstrap only upgrade Mercurial if the
version is older than MODERN_MERCURIAL_VERSION.
As a side effect, this avoids upgrading to version 5.0.1, which doesn't
come with wheels at the moment.
Differential Revision: https://phabricator.services.mozilla.com/D35754
--HG--
extra : moz-landing-system : lando
The `mach resource-usage` page supports a list of build resource data,
but only ever displays the first one. Nothing actually creates a list
with multiple items automatically, but one might want to do that
manually to explore data from multiple builds more conveniently.
So if such a list exists, we expose a dropdown list of all the data
available, and switch the graph when a different item is chosen from the
list.
Differential Revision: https://phabricator.services.mozilla.com/D35757
--HG--
extra : moz-landing-system : lando
The git version shipped in some versions of OSX is patched by apple in a
way such that doing `git push hg::ssh://...` fails with an error message
like `Invalid remote name "hg::ssh://...`.
So instead, we define a named remote via inline configuration, and use
that remote's name for the push.
Differential Revision: https://phabricator.services.mozilla.com/D35351
--HG--
extra : moz-landing-system : lando
The git version shipped in some versions of OSX is patched by apple in a
way such that doing `git push hg::ssh://...` fails with an error message
like `Invalid remote name "hg::ssh://...`.
So instead, we define a named remote via inline configuration, and use
that remote's name for the push.
Differential Revision: https://phabricator.services.mozilla.com/D35351
--HG--
extra : moz-landing-system : lando
This makes running without mach more consistent. e.g. running
`make -C $objdir/toolkit/library/rust target` makes the cargo log
verbose, and adding `-s` makes it less verbose.
Differential Revision: https://phabricator.services.mozilla.com/D35521
--HG--
extra : moz-landing-system : lando
When the `MOZ_RUST_TIER` environment variable is set, we enable a separate
tier that builds rust code only. This is useful to measure build times for
rust code separately from other compilations.
Differential Revision: https://phabricator.services.mozilla.com/D35501
--HG--
extra : moz-landing-system : lando
NSS_ALLOW_SSLKEYLOGFILE no longer has issues upstream, we can allow it again.
Differential Revision: https://phabricator.services.mozilla.com/D34915
--HG--
extra : moz-landing-system : lando
Bug 1394825 bumped the minimum version of clang we use on automation,
for the base toolchain jobs, and there's a libclang test for bindgen,
but we should reject the compiler in the first place.
The check to do that works in both C and C++, contrary to the original
3.6 test that was checking a C++ feature ; the test was later changed
for 3.9, using a check that would have worked in both C and C++, but
the C exception was left around. We remove it now.
Differential Revision: https://phabricator.services.mozilla.com/D34083
--HG--
extra : moz-landing-system : lando
This works around the following situation: the Ngv/fat AAR task is
deterministic, but it depends on N tasks for multiple architectures,
adn the exact l10n changeset pulled by each N task depends on the tip
of the l10n HG repository at the time the N task executes.
For now, just ignore the obvious localized resources: *.ftl, *.dtd,
*.properties.
Differential Revision: https://phabricator.services.mozilla.com/D33413
--HG--
extra : moz-landing-system : lando
Not sure why this wasn't done before, but it prevented me from running `./mach rust vendor`.
Differential Revision: https://phabricator.services.mozilla.com/D34127
--HG--
extra : moz-landing-system : lando
Skips over the incremental cache when performing a clobber. The incremental compilation cache is located at:
`$(objdir)/$(rust_target)/$(rust_build_kind)/incremental`
When cross compiling there can be two caches, one for the host and one for the target so we handle both.
Differential Revision: https://phabricator.services.mozilla.com/D31310
--HG--
extra : moz-landing-system : lando
Pass in the substs dictionary to `Clobberer` so that we can use it to query `cargo` paths.
Differential Revision: https://phabricator.services.mozilla.com/D31309
--HG--
extra : moz-landing-system : lando
Pull out the logic for filtering subdirectories and deleting them into
reusable functions.
Differential Revision: https://phabricator.services.mozilla.com/D31308
--HG--
extra : moz-landing-system : lando
Use the same logic as for --with-android-sdk to use the NDK downloaded
by mach bootstrap by default.
Differential Revision: https://phabricator.services.mozilla.com/D34251
--HG--
extra : moz-landing-system : lando
After bug 1519954, the build system will pick the compiler in
~/.mozbuild on its own, so there's no need to suggest people to manually
put that in their mozconfig.
Differential Revision: https://phabricator.services.mozilla.com/D33903
--HG--
extra : moz-landing-system : lando
As established previously, generated source files that are not included
don't need to be built during the export tier.
Differential Revision: https://phabricator.services.mozilla.com/D33771
--HG--
extra : moz-landing-system : lando
Pull out the logic for filtering subdirectories and deleting them into
reusable functions.
Differential Revision: https://phabricator.services.mozilla.com/D31308
--HG--
extra : moz-landing-system : lando
In a branch for `isinstance(obj, (HostSources, HostGeneratedSources))`,
`isinstance(obj, GeneratedSources)` is clearly not going to match
anything. `isinstance(obj, HostGeneratedSources)` is what was intended.
Differential Revision: https://phabricator.services.mozilla.com/D33603
--HG--
extra : moz-landing-system : lando
webidlsrcs.mk is treated specially by the build system, and this entire
process should be supported better by moz.build, but in the meantime this
fixes the clobber bug caused by not re-building these targets when defines
are updated.
Differential Revision: https://phabricator.services.mozilla.com/D33415
--HG--
extra : moz-landing-system : lando
Invoking MozbuildObject.substs causes an exception to be raised if there is no
BuildEnvironment. This means that we hit an error on |mach help| if there is no
build, which is not ideal for people trying to learn how to build :).
Differential Revision: https://phabricator.services.mozilla.com/D33217
--HG--
extra : moz-landing-system : lando
This follows the model set down for EME artifacts:
- a new tier is added that uses a new Python build action to fetch and
artifacts
- the action unpacks the fetched artifacts and moves specific inputs
into places expected by the build and packager
- in automation, MOZ_ARTIFACT_TASK* is used to ensure the artifacts
come from the correct tasks
In this case, the artifact fetching is done entirely in a new Python
build action that internally uses `mach artifact install --job ...`.
The action also verifies that the fetched artifacts are compatible and
that we're not assembling a fat AAR that is nonsensical. The specific
inputs are not used in the Fennec APK that is produced; they're only
used in the GeckoView AAR that is produced.
The artifact fetching itself required tweaking to fetch only
`target.maven.zip` artifacts and to not unpack them.
The specific inputs used are the native libraries (libs/$ARCH/*.so)
and the architecture-specific preference files ($ARCH/greprefs.js and
defaults/pref/$ARCH/geckoview-prefs.js). None of these inputs are
impacted by l10n.
Differential Revision: https://phabricator.services.mozilla.com/D31572
--HG--
extra : moz-landing-system : lando
This allows to use the existing artifacts VCS-based crawling to
download the "raw" target.maven.zip from Android jobs and not process
it further. It's just put in a specific directory, ready for use.
This isn't a big deal in automation, where all URLs are known, but
it's very useful when building locally and the VCS and the pushlog
must be consulted to determine task URLs.
Differential Revision: https://phabricator.services.mozilla.com/D31571
--HG--
extra : moz-landing-system : lando
(adds __build_class__ to the builtin sandbox and allow CombinedDependsFunction to be hashed)
Differential Revision: https://phabricator.services.mozilla.com/D28114
--HG--
extra : moz-landing-system : lando
shellutil.py needed special handling since it is invoked earlier than the
vendor directory adding we do, so it does not have six available.
Differential Revision: https://phabricator.services.mozilla.com/D28096
--HG--
extra : moz-landing-system : lando
Lint python/mozbuild/{mozbuild,mozpack}.
This makes sure we use byte strings (type(b'')) in many places.
This also has a few places where we know we want unicode, by enforcing it.
This code also has a few places where we call into windows API's which expect either all args to be unicode,
or all args to be bytestrings (not-unicode) so make sure those calls align.
This is the most risky of the stack, since there are some mach commands and codepaths that are neither excercised in automation
nor excercised in tests.
Differential Revision: https://phabricator.services.mozilla.com/D26645
--HG--
extra : moz-landing-system : lando
Lint python/mozbuild/{mozbuild,mozpack}. r=#build
Add 'from __future__ import absolute_import' to all missing files in python/mozbuild/{mozbuild,mozpack}/**/*.py
Differential Revision: https://phabricator.services.mozilla.com/D26643
--HG--
extra : moz-landing-system : lando
Leaving one fix for an explicit review.
Lint python/mozbuild/{mozbuild,mozpack}. r=#build
Differential Revision: https://phabricator.services.mozilla.com/D26641
--HG--
extra : moz-landing-system : lando
Lint python/mozbuild/{mozbuild,mozpack}.
ran './mach lint -l py2 -l flake8 -l shellcheck -l codespell -l yaml python/mozbuild/{mozbuild,mozpack}/ --fix' in order
to undo some black changes and get closer to making this folder able to be validated on every lint run
Differential Revision: https://phabricator.services.mozilla.com/D26640
--HG--
extra : moz-landing-system : lando
Moritz Birghan
Bryce Seager van Dyk
Joel Maher
Andrew Halberstadt
Johan Lorenzo
Bastien Abadie
Mike Hommey
Andi-Bogdan Postelnicu
James Willcox
Edwin Gao
Makoto Kato
Mike Shal
Simon Sapin
Agi Sferro
Chris Manchester
Tom Prince
Victor Porof
Andreea Pavel
Justin Wood
Ed Lee
Francesco Lodolo (:flod)
Ryan VanderMeulen
Jeane Carlos
Sylvestre Ledru
Samuel Thibault
Kartikaya Gupta
championshuttler
Cosmin Sabou
Rok Garbas
Dorel Luca
prathiksha
L. David Baron
Nick Alexander
Eric Rahm
Bob Clary
David Teller
Henri Sivonen
shindli
Jorg K
Jeremy Lempereur