When multiple java versions are in use, some OSes have a mechanism to
change the "default"/system Java.
This can cause unexpected build failures if the system Java changes.
So, in bootstrap, if the Java version found is valid, then its path
should be encoded into mozconfig.
Differential Revision: https://phabricator.services.mozilla.com/D90163
The idea here was that we keep the NDK's downloaded in this directory as a "cache", such that if the download gets interrupted, then we can resume the download from an earlier point. This logic was walked back by bug 1637379, which deletes the cache.
In the spirit of the original version of the code, remove the `os.rmdir()` that was introduced by that bug.
An alternative fix for this is to download the NDK to a temporary directory and just clean that up entirely after every call to `mach bootstrap`, but then we would be forgoing the build cache behavior, which I'm not sure makes sense at this point in time.
Differential Revision: https://phabricator.services.mozilla.com/D90125
In bootstrap, pacman should ensure its database is up-to-date.
Otherwise, when installing packages, pacman will slowly query every
mirror when attempting to download an out-of-date package.
Differential Revision: https://phabricator.services.mozilla.com/D89958
`bootstrap` is written in such a way that we don't necessarily assume the existence of either the global state directory (`~/.mozbuild`) or a local checkout of `mozilla-central`. The independence from `~/.mozbuild` is a design decision that may have been appropriate at some point, but today it's arguably useless to continue executing `bootstrap` without a global state directory (we install a bunch of build dependencies there, as well as the `mach` `virtualenv`s that are needed for running non-`bootstrap` `mach` commands after bug 1656993). The independence from a local checkout is an artifact of the old design of `python/mozboot/bin/bootstrap.py`, which is no longer necessary as of bug 1647792.
With that in mind, 1) throw errors if we can't create the global state directory or find the topsrcdir, and 2) remove all existing conditionals of the form "if the global state directory exists" or "if the root of the checkout exists" since these conditions are always going to be true in practice.
Differential Revision: https://phabricator.services.mozilla.com/D89220
This will be found in a couple places, so we might as well make a helper function. For symmetry put it in the same file where we keep the helper function to locate the `state_dir`.
Differential Revision: https://phabricator.services.mozilla.com/D89156
`mach artifact` has a dependency on `zstandard`, which is installed in the `mach` `virtualenv`s, so we have to run `mach artifact` with the correct `virtualenv`. Also create the `virtualenv`s earlier in the process to account for this.
This all has a dependency on the existence of a checkout (which has the `mach` script with all its dependencies on everything else), but after bug 1647792 that's not a concern.
Differential Revision: https://phabricator.services.mozilla.com/D87920
This consolidates the `have_clone` logic in one place unconditionally. After bug 1647792 we're deprecating the use case where `bootstrap` is run without a clone, so that's not a problem.
In reality the whole `have_clone` thing isn't necessary any more (`have_clone` is always going to be `True` in practice), but I'll save that for a bigger refactoring.
Differential Revision: https://phabricator.services.mozilla.com/D89152
Note that when I refer to "standalone `bootstrap.py`" here, I'm referring to the file `python/mozboot/bin/bootstrap.py` and no other similarly-named file in-tree.
The current design, where standalone `bootstrap.py` downloads a small portion of the `mozilla-central` repo and then works through all the `bootstrap` logic, performing a clone in the middle of the `bootstrap` process, has some deficiencies, namely:
1. Refactoring code that is shared between the `bootstrap` logic and the mainline `mach` logic is, if not impossible, more difficult than it needs to be, since standalone `bootstrap.py` needs to download a set of files that bootstraps a build environment perfectly with no other dependencies in `mozilla-central`. As a result we have some [duplicated or redundant code](https://searchfox.org/mozilla-central/rev/c6676771df58c6e0098574bc6b11517acbf264cf/python/mozboot/mozboot/base.py#349) and some stuff that has been [refactored into the `mozboot` directory](https://searchfox.org/mozilla-central/source/python/mozboot/mozboot/util.py) irrespective of whether it actually makes sense to go there (see bug 1649850).
2. Since `mach bootstrap` has access to the entire `mozilla-central` environment, but the (much less frequently exercised) standalone `bootstrap.py` script does *not*, this can lead people to write patches that work fine in `mach bootstrap` but which regress standalone `bootstrap.py`. Furthermore, testing `bootstrap` patches with standalone `bootstrap.py` is difficult. So this is a not infrequent source of regressions; bugs 1652736 and 1643158 are recent examples. Furthermore, typically these regressions are "fixed" by adding more code duplication or by replacing battle-tested frequently-used libraries (either `m-c`-internal or third-party) with less robust bespoke code.
3. Issue (2) is avoidable if people are sufficiently critical during code review, but at *best*, this is a weird extra level of mental overhead that we need to keep in mind only for `bootstrap` patches.
This patch preserves back-compatibility and the validity of existing documentation by factoring out all the logic to clone `mozilla-central` into standalone `bootstrap.py` directly, and cloning *before* calling into `mach bootstrap` directly. This gives us only one official entry point into the bootstrapper, namely, `mach bootstrap`.
There are a couple concrete implications of this change:
1. Now, the clone happens before any other interesting work happens, so people may have to wait ~an hour before actually beginning to engage with the `bootstrap` wizard. While it's arguably slightly less convenient, I'm not sure it matters enough that we should block this patch or a similar one on it.
2. The `hg`/`git` configuration step now happens *after* the clone rather than before it. Looking at what the `hg` and `git` configuration wizards actually do today, I don't think this matters either (all of the configurations can easily happen after cloning the repo).
Another note: `bootstrap` installs `git-cinnabar` to the `.mozbuild` state directory. We could have duplicated all of that logic over to standalone `bootstrap.py`, but it's non-trivial and I didn't think that made any sense, so instead we have standalone `bootstrap.py` download a temporary version and use it for the initial clone if necessary.
Differential Revision: https://phabricator.services.mozilla.com/D85058
This brings it up-to-date with the minimum allowable version in `version-control-tools`; these two versions should be equal to each other, as the [comment in the source code](4b0de666d1/hgext/configwizard/__init__.py (l59)) suggests. Also add a similar comment in this file to prevent the likelihood of a mismatch going into the future.
Differential Revision: https://phabricator.services.mozilla.com/D89022
In two different places we've been encountering issues regarding 1) how we configure the system Python environment and 2) how the system Python environment relates to the `virtualenv`s that we use for building, testing, and other dev tasks. Specifically:
1. With the push to use `glean` for telemetry in `mach`, we are requiring (or rather, strongly encouraging) the `glean_sdk` Python package to be installed with bug 1651424. `mach bootstrap` upgrades the library using your system Python 3 in bug 1654607. We can't vendor it due to the package containing native code. Since we generally vendor all code required for `mach` to function, requiring that the system Python be configured with a certain version of `glean` is an unfortunate change.
2. The build uses the vendored `glean_parser` for a number of build tasks. Since the vendored `glean_parser` conflicts with the globally-installed `glean_sdk` package, we had to add special ad-hoc handling to allow us to circumvent this conflict in bug 1655781.
3. We begin to rely more and more on the `zstandard` package during build tasks, this package again being one that we can't vendor due to containing native code. Bug 1654994 contained more ad-hoc code which subprocesses out from the build system's `virtualenv` to the SYSTEM `python3` binary, assuming that the system `python3` has `zstandard` installed.
As we rely more on `glean_sdk`, `zstandard`, and other packages that are not vendorable, we need to settle on a standard model for how `mach`, the build process, and other `mach` commands that may make their own `virtualenv`s work in the presence of unvendorable packages.
With that in mind, this patch does all the following:
1. Separate out the `mach` `virtualenv_packages` from the in-build `virtualenv_packages`. Refactor the common stuff into `common_virtualenv_packages.txt`. Add functionality to the `virtualenv_packages` manifest parsing to allow the build `virtualenv` to "inherit" from the parent by pointing to the parent's `site-packages`. The `in-virtualenv` feature from bug 1655781 is no longer necessary, so delete it.
2. Add code to `bootstrap`, as well as a new `mach` command `create-mach-environment` to create `virtualenv`s in `~/.mozbuild`.
3. Add code to `mach` to dispatch either to the in-`~/.mozbuild` `virtualenv`s (or to the system Python 3 for commands which cannot run in the `virtualenv`s, namely `bootstrap` and `create-mach-environment`).
4. Remove the "add global argument" feature from `mach`. It isn't used and conflicts with (3).
5. Remove the `--print-command` feature from `mach` which is obsoleted by these changes.
This has the effect of allowing us to install packages that cannot be vendored into a "common" place (namely the global `~/.mozbuild` `virtualenv`s) and use those from the build without requiring us to hit the network. Miscellaneous implementation notes:
1. We allow users to force running `mach` with the system Python if they like. For now it doesn't make any sense to require 100% of people to create these `virtualenv`s when they're allowed to continue on with the old behavior if they like. We also skip this in CI.
2. We needed to duplicate the global-argument logic into the `mach` script to allow for the dispatch behavior. This is something we avoided with the Python 2 -> Python 3 migration with the `--print-command` feature, justifying its use by saying it was only temporarily required until all `mach` commands were running with Python 3. With this change, we'll need to be able to determine the `mach` command from the shell script for the forseeable future, and committing to this forever with the cost that `--print-command` incurs (namely `mach` startup time, an additional .4s on my machine) didn't seem worth it to me. It's not a ton of duplicated code.
Differential Revision: https://phabricator.services.mozilla.com/D85916
There's a Windows Defender CLI (`Get-MpComputerStatus`) available, but MozillaBuild
can't (easily) access PowerShell. So, instead, we find Windows Defender status and path exclusions
by checking the registry at HKLM\SOFTWARE\Microsoft\Windows Defender.
Determining if Windows Defender is "on" or not is a surprisingly vague task.
Not only does it encompass a lot of components (of which Ricky and I believe "Real-time protection"
is the part slowing down the build), but there's (at least?) two different kinds of "disabled"
states that it can be in:
* If "disabled" via Settings, it will turn itself back on after a reboot
* If an antivirus is installed, it will turn itself off permanently
Unfortunately, disabling "Real-time protection" in Settings doesn't affect any registry keys, but
I'm opting to ignore this because I doubt many users are manually disabling this protection before
each build. The repercussion of this is that users may be incorrectly warned "your Firefox
directory isn't excluded from Windows Defender!".
Focusing on the antivirus use case and querying the registry I found that:
* Querying the `IsServiceRunning` value wasn't sufficient because new Windows installations didn't have that key
* Querying the `DisableRealtimeMonitoring` value was inconsistent - it can be missing, and not always because an antivirus removed it
* Querying the `DisableAntiVirus` value doesn't _sound_ accurate (we care about Real-time Protection), but it's consistently "off" for non-AV machines, and "on" for machines with an AV installed. So, this is our winner!
TL;DR: there may be some "false positive" warnings about excluding the Firefox srcdir, but they're
accurate for my test cases and workaround-able (just add the exclusion to Windows Defender).
Also, this patch updates our Windows Defender docs to make them:
* More easily link-able
* Have direct advice to resolve the issue
Differential Revision: https://phabricator.services.mozilla.com/D85952
Today we don't require that `mach` `CommandProvider`s subclass from any particular parent class and we're very lax about the requirements they must meet. While that's convenient in certain circumstances, it has some unfortunate implications for feature development.
Today the only requirements that we have for `CommandProvider`s are that they have an `__init__()` method that takes either 1 or 2 arguments, the second of which must be called `context` and is populated with the `mach` `CommandContext`. Again, while this flexibility is occasionally convenient, it is limiting. As we add features to `mach`, having a better idea what the shape of our `CommandProvider`s are and how we can instantiate them and use them is increasingly important, and this gives us additional control when having `mach` configure `CommandProvider`s based on data that is only available at the `mach` level. In particular, we plan to leverage this in bugs 985141 and 1654074.
Here we add validation to the `CommandProvider` decorator to ensure all classes inherit from `MachCommandBase`, update all `CommandProvider`s in-tree to inherit from `MachCommandBase`, and update source and test code accordingly.
Follow-up work: we now require (de facto) that the `context` be populated with a `topdir` attribute by the `populate_context_handler` function, since instantiating the `MachCommandBase` requires a `topdir` be provided. This is fine for now in the interest of keeping this patch reasonably sized, but some additional refactoring could make this cleaner.
Differential Revision: https://phabricator.services.mozilla.com/D86255
A newer version of Glean is needed to take advantage of the recent "optional version" change.
Pins glean to 31.5.0 to avoid breaking changes, such as to the YAML schema.
Differential Revision: https://phabricator.services.mozilla.com/D84566
Check whether Python 3 is up-to-date and bail out with specific installation guidance if not. Don't install or upgrade Python 2/3. Also check whether Python 2 is up-to-date, and while we don't block `bootstrap` on it (because Python 2 isn't required to build), also supplement with specific installation guidance for people who need it.
Differential Revision: https://phabricator.services.mozilla.com/D82739
Also delete a bunch of dead code that was apparently meant to be useful for subclassing, but there's no subclasses of `DebianBootstrapper` in tree.
Differential Revision: https://phabricator.services.mozilla.com/D82073
Installs glean to the currently-active python installation during bootstrap, whether
that be a virtualenv or the system instance of python.
Differential Revision: https://phabricator.services.mozilla.com/D83012
If we're going to ditch the old breakpad dump-syms from the build, people
are going to need have this locally if they ever want to build packages,
etc.
Differential Revision: https://phabricator.services.mozilla.com/D83150
If we're going to ditch the old breakpad dump-syms from the build, people
are going to need have this locally if they ever want to build packages,
etc.
Differential Revision: https://phabricator.services.mozilla.com/D83150
This is where the `compare-mozconfig` test properly belongs, and `mozbuild` tests already re-run every time a `mozboot` file changes.
Differential Revision: https://phabricator.services.mozilla.com/D83130
Since bug 1647865, we're no longer allowing `bootstrap` to be run with Python 2. Therefore we have leeway to make a bunch of simplifications.
Differential Revision: https://phabricator.services.mozilla.com/D82737
Prior to this patch JAVA_HOME was appended to path and then the first instance
of java on those paths was used and checked for version compatibility. This
meant that if an incompatible version on java is on the path JAVA_HOME could not
be used to point to a different, compatible version.
Following this patch, JAVA_HOME can be used as a more fine grained selector of
java. Only if JAVA_HOME is not set will we check the path. To be clear, even if
JAVA_HOME is set incorrectly, we will not fall back to path -- only if JAVA_HOME
is not set will we try and use the path.
Differential Revision: https://phabricator.services.mozilla.com/D82418
D79393 introduced a new parameter to `configure_git()` but there is a remaining call that does not pass it. This patch fixes it.
Differential Revision: https://phabricator.services.mozilla.com/D82129
When moving from mozreview to Phabricator, the old mozreview git hook
had to be removed. Logic to automate the removal of this hook was added
to vcs-setup.
This commit removes this automated removal since it has been 2 years
since it landed.
Differential Revision: https://phabricator.services.mozilla.com/D82125
We get a lot of bugs about building Firefox for Android not working, and there's been ongoing confusion about whether this is something that does work or who has ever worked. The warning should hopefully clarify the current situation for people.
Differential Revision: https://phabricator.services.mozilla.com/D81585
This message dates from bug 1177128, the commit where this bootstrapper was added. At the time of writing that was certainly true, but today it doesn't appear to be true in any meaningful way. There's certainly an expectation that installing MozillaBuild and running `mach bootstrap` will get your machine ready to build Firefox; or at least, if it doesn't, it prints out some error messages telling you to do extra stuff, like we see in bug 1644794 for example. Therefore, I don't see how this message is relevant at all any more.
Differential Revision: https://phabricator.services.mozilla.com/D80730
This uses sys.exit() instead of throwing an exception since:
1. It isn't an error, and bug 1646406 isn't finished yet
2. The output is more straightforward for the user
This verification happens separately from the other python-verification logic, but that's
because I wanted the validation to happen as soon as possible - before the user invests too
much time, and before too much of the script ran with the wrong interpreter.
Differential Revision: https://phabricator.services.mozilla.com/D80726
Additionally, recommends the installation of AdoptOpenJDK instead of the Oracle JDK.
This required a different "JAVA_HOME" detection mechanism, which is implemented in this revision.
Differential Revision: https://phabricator.services.mozilla.com/D79547
Previously, python import syntax was used to refer users to the definition of the --application-choice arguments.
However, it isn't straightforward to the uninitiated, especially since it doesn't work in searchfox.
Instead, by providing a file reference, it should be more accessible.
Differential Revision: https://phabricator.services.mozilla.com/D79003
Rather than always printing instructions at the end of the bootstrap phase, we will now create a mozconfig
file if one doesn't exist and there's configuration to be written.
Differential Revision: https://phabricator.services.mozilla.com/D78417
Toolchains that are used for local development need to be built on a level-3
branch to installable via `mach bootstrap`. Add an attribute to track the fact
that a toolchain is used that way, and:
- ensure that everything installed via `mach boostrap` has that attribute set
- ensure that everything with that attribute set is built on trunk projects
We could additionally verify that attribute is only set on things used by
bootstrap, but bootstrap doesn't currently have an exhaustive list of things
that it might install, making that difficult.
Differential Revision: https://phabricator.services.mozilla.com/D77706
Ricky Stewart
Mitchell Hentges
Mark Hammond
Nathan Froyd
Cosmin Sabou
Sebastian Hengst
Bryce
Kagami Sascha Rosylight
Karl Dubost
David Major
Valentin Gosu
Tom Prince