In bootstrap, pacman should ensure its database is up-to-date.
Otherwise, when installing packages, pacman will slowly query every
mirror when attempting to download an out-of-date package.
Differential Revision: https://phabricator.services.mozilla.com/D89958
In addition to the existing build telemetry, also gather the stats and
report with Glean. This new telemetry is reported in tandem with the existing
telemetry to allow testing and confidence before a full roll-out.
Additionally, Glean isn't compatible with Python 2, so the new telemetry only runs
on Python 3 mach commands.
Differential Revision: https://phabricator.services.mozilla.com/D83572
This was originally meant to allow `virtualenv`s to use packages from a parent Python environment without having to re-install them. This turned out to not pan out as we would have liked, so we're going another way to solve the same problem. Bug 1660351 walked back a bunch of this logic; this patch deletes the rest of it.
Differential Revision: https://phabricator.services.mozilla.com/D89492
This commit does the following.
- Renames `slashslash` as `dumbComments`. As a result, it now comes before
`emptyLines` in alphabetical ordering, which means that if you apply both
`dumbComments` and `emptyLines`, lines that contain only comments will be
fully removed.
(I contemplated changing the filter ordering to match the order specified,
rather than using alphabetical ordering, but that was more invasive and not
obviously better.)
- Changes `dumbComments` so it only applies if the comment is at the start of
the line (with optional leading whitespace). This is so it can be used with
prefs files, which contain lines like `pref("foo", "https://mozilla.org");`
where the `//` must not be treated as a comment.
Note that `slashslash` wasn't being used anywhere.
Depends on D88240
Differential Revision: https://phabricator.services.mozilla.com/D88242
It's not used, probably because it's pretty strange and hard to imagine using
safely. (Stripping leading and trailing space could be useful, but collapsing
sequences of spaces? Hmm.)
Differential Revision: https://phabricator.services.mozilla.com/D88240
The llvm-strip from clang-11 complains about this file. This file doesn't really interest us anyway -- it's imported from elsewhere -- so just avoid it.
Differential Revision: https://phabricator.services.mozilla.com/D89491
`bootstrap` is written in such a way that we don't necessarily assume the existence of either the global state directory (`~/.mozbuild`) or a local checkout of `mozilla-central`. The independence from `~/.mozbuild` is a design decision that may have been appropriate at some point, but today it's arguably useless to continue executing `bootstrap` without a global state directory (we install a bunch of build dependencies there, as well as the `mach` `virtualenv`s that are needed for running non-`bootstrap` `mach` commands after bug 1656993). The independence from a local checkout is an artifact of the old design of `python/mozboot/bin/bootstrap.py`, which is no longer necessary as of bug 1647792.
With that in mind, 1) throw errors if we can't create the global state directory or find the topsrcdir, and 2) remove all existing conditionals of the form "if the global state directory exists" or "if the root of the checkout exists" since these conditions are always going to be true in practice.
Differential Revision: https://phabricator.services.mozilla.com/D89220
This will be found in a couple places, so we might as well make a helper function. For symmetry put it in the same file where we keep the helper function to locate the `state_dir`.
Differential Revision: https://phabricator.services.mozilla.com/D89156
Bug 1659539 caused the unit tests of this class to suddenly start running on Linux; it failed with a type error that suggests this test has never really properly worked, at least not with the version of Mercurial we're using in CI (`unsupported changeid '0' of type <type 'unicode'>`). The class itself isn't used anywhere besides these tests, so just delete the entire class.
Differential Revision: https://phabricator.services.mozilla.com/D89205
The manifest file hasn't actually done anything since XPT definitions were
moved to the libxul binary, and now just generates warnings in local builes.
Differential Revision: https://phabricator.services.mozilla.com/D89197
`mach artifact` has a dependency on `zstandard`, which is installed in the `mach` `virtualenv`s, so we have to run `mach artifact` with the correct `virtualenv`. Also create the `virtualenv`s earlier in the process to account for this.
This all has a dependency on the existence of a checkout (which has the `mach` script with all its dependencies on everything else), but after bug 1647792 that's not a concern.
Differential Revision: https://phabricator.services.mozilla.com/D87920
The ability to get the path to the Python executable from a given `virtualenv` location is generally useful outside the context of all the extra stuff a `VirtualenvManager` provides, so refactor it out into a lighter-weight helper class.
Differential Revision: https://phabricator.services.mozilla.com/D89175
Pipenv is heavy weight and overkill for the purposes it is being used. We'd like to remove it from the tree and |mach python-test| was one of the last remanining use cases.
Remove the `--python` command-line argument as a result. Users who wish to run unit tests with Python 2 can do `MACH_PY2=1 ./mach python-test ...` or `python2 ./mach python-test ...`.
Also update a few unit tests that would break otherwise in the presence of this change.
There were a couple lines in the `setup.py` for `mozlog` that were problematic for tests and was resulting in errors due to the `mozlog` plugin being loaded by `pytest` more than once. We just delete those lines and bump up the major version number of the package to fix it.
Differential Revision: https://phabricator.services.mozilla.com/D88296
Otherwise `gyp` can choose incorrectly when trying to figure out which Python to use to run its internal scripts, which can manifest as test failures in certain circumstances.
Differential Revision: https://phabricator.services.mozilla.com/D89165
This consolidates the `have_clone` logic in one place unconditionally. After bug 1647792 we're deprecating the use case where `bootstrap` is run without a clone, so that's not a problem.
In reality the whole `have_clone` thing isn't necessary any more (`have_clone` is always going to be `True` in practice), but I'll save that for a bigger refactoring.
Differential Revision: https://phabricator.services.mozilla.com/D89152
Note that when I refer to "standalone `bootstrap.py`" here, I'm referring to the file `python/mozboot/bin/bootstrap.py` and no other similarly-named file in-tree.
The current design, where standalone `bootstrap.py` downloads a small portion of the `mozilla-central` repo and then works through all the `bootstrap` logic, performing a clone in the middle of the `bootstrap` process, has some deficiencies, namely:
1. Refactoring code that is shared between the `bootstrap` logic and the mainline `mach` logic is, if not impossible, more difficult than it needs to be, since standalone `bootstrap.py` needs to download a set of files that bootstraps a build environment perfectly with no other dependencies in `mozilla-central`. As a result we have some [duplicated or redundant code](https://searchfox.org/mozilla-central/rev/c6676771df58c6e0098574bc6b11517acbf264cf/python/mozboot/mozboot/base.py#349) and some stuff that has been [refactored into the `mozboot` directory](https://searchfox.org/mozilla-central/source/python/mozboot/mozboot/util.py) irrespective of whether it actually makes sense to go there (see bug 1649850).
2. Since `mach bootstrap` has access to the entire `mozilla-central` environment, but the (much less frequently exercised) standalone `bootstrap.py` script does *not*, this can lead people to write patches that work fine in `mach bootstrap` but which regress standalone `bootstrap.py`. Furthermore, testing `bootstrap` patches with standalone `bootstrap.py` is difficult. So this is a not infrequent source of regressions; bugs 1652736 and 1643158 are recent examples. Furthermore, typically these regressions are "fixed" by adding more code duplication or by replacing battle-tested frequently-used libraries (either `m-c`-internal or third-party) with less robust bespoke code.
3. Issue (2) is avoidable if people are sufficiently critical during code review, but at *best*, this is a weird extra level of mental overhead that we need to keep in mind only for `bootstrap` patches.
This patch preserves back-compatibility and the validity of existing documentation by factoring out all the logic to clone `mozilla-central` into standalone `bootstrap.py` directly, and cloning *before* calling into `mach bootstrap` directly. This gives us only one official entry point into the bootstrapper, namely, `mach bootstrap`.
There are a couple concrete implications of this change:
1. Now, the clone happens before any other interesting work happens, so people may have to wait ~an hour before actually beginning to engage with the `bootstrap` wizard. While it's arguably slightly less convenient, I'm not sure it matters enough that we should block this patch or a similar one on it.
2. The `hg`/`git` configuration step now happens *after* the clone rather than before it. Looking at what the `hg` and `git` configuration wizards actually do today, I don't think this matters either (all of the configurations can easily happen after cloning the repo).
Another note: `bootstrap` installs `git-cinnabar` to the `.mozbuild` state directory. We could have duplicated all of that logic over to standalone `bootstrap.py`, but it's non-trivial and I didn't think that made any sense, so instead we have standalone `bootstrap.py` download a temporary version and use it for the initial clone if necessary.
Differential Revision: https://phabricator.services.mozilla.com/D85058
This brings it up-to-date with the minimum allowable version in `version-control-tools`; these two versions should be equal to each other, as the [comment in the source code](4b0de666d1/hgext/configwizard/__init__.py (l59)) suggests. Also add a similar comment in this file to prevent the likelihood of a mismatch going into the future.
Differential Revision: https://phabricator.services.mozilla.com/D89022
This logic is meant to expose packages from a globally-installed Python to be used by the in-`objdir` `virtualenv`s, so for example we don't have to figure out how to install `zstandard` (or other Python packages with native code that may or may not have prebuilt wheels for any given platform) in those `virtualenv`s. Bug 1660351 augmented that logic to work within the requirements of bug 1660353. This worked mostly, but is causing builds to unconditionally break on Arch Linux, caused a couple test failures, and in general is just introducing other weird behaviors downstream, and issues with the resultant `PYTHONPATH`s are hard to diagnose and fix.
In the long-term we'll have to permanently solve the `zstandard` problem and pave the way for other Python packages with native code as well, but that's not an urgent need.
The ultimate goal is to completely remove `inherit-from-parent-environment`, but we can't do that until bug 1659539 is solved.
Partially reverts bugs 1660351. Entirely reverts bug 1660353, restoring that file to as it was before that patch.
Differential Revision: https://phabricator.services.mozilla.com/D89001
This patch adds two new options to mozperftest. The --simplify-names argument can be provided to enable simplification of metric names and the --simplify-exclude option allows users to select which metrics to skip in the simplification. A bug relating to setting the default of a list option is also fixed here.
Differential Revision: https://phabricator.services.mozilla.com/D88917
This fixes a local failure in mozperftest by mocking the find_node_executable function in the visual-metrics tests. It is not needed for those tests.
Depends on D88915
Differential Revision: https://phabricator.services.mozilla.com/D88916
Improves glean performance.
Prior to this change, using Glean adds ~500ms to each `mach` run.
After this change, using Glean adds ~200ms to each `mach` run.
Differential Revision: https://phabricator.services.mozilla.com/D88691
Gijs Kruitbosch
Razvan Maries
Mark Banner
Mitchell Hentges
Tarek Ziadé
Narcis Beleuzu
Mihai Alexandru Michis
Csoregi Natalia
Ricky Stewart
Myeongjun Go
Mike Hommey
Nicholas Nethercote
David Major
Andrew Halberstadt
Sylvestre Ledru
Kris Maglione
Yue Zhang
Cristina Coroiu
Florian Quèze
Gregory Mierzwinski