`commands.dispatch(...)` don't respect command argument defaults set up
in decorators. Instead, it falls back to the python parameter default,
which is essentially unset.
The solution in this case is to duplicate the `default` value.
Differential Revision: https://phabricator.services.mozilla.com/D125605
Adds a `MozVirtualenvMetadata` file abstraction that allows us to:
* Statically determine if a virtualenv is managed by Mach.
* Determine the virtualenv's name.
Note: I called this file `moz_virtualenv_metadata.json` instead of
`mach_virtualenv_metadata.json` because I didn't want to be ambiguous
between "the Mach virtualenv" and "a Mach-managed virtualenv".
Differential Revision: https://phabricator.services.mozilla.com/D124517
Tweak the `VirtualenvManager` API to accept a single base "python"
executable during initialization, then to consistently use it for
up-to-date checking, construction, etc.
This constraint allows future simplification for behaviour involving
the "base" python.
Differential Revision: https://phabricator.services.mozilla.com/D124516
When porting commands to have their own virtualenv,
it's useful to have helpful warnings when the associated
requirements definition file is missing.
Differential Revision: https://phabricator.services.mozilla.com/D122893
Maps virtualenvs one-to-one with their associated requirements
definition.
For example:
```
Name: "docs"
Virtualenv location: <snip>/_virtualenvs/docs
Requirements location: $topsrcdir/build/docs_virtualenv_packages.txt
```
An issue to be resolved in the future is that it's tricky to know that,
when you define a new virtualenv, you have to *know* that a requirements
definition needs to exist in `build/`.
As part of this change, the default virtualenv ("common") was
split from the build virtualenv ("build").
This required changes to the python-test virtualenv since
python-tests depend on `glean_parser`, but were getting it
implicitly from the "build" virtualenv's requirements.
This addition to the `python-test` virtualenv is temporary and
will be removed when bug 1724273 is resolved.
Differential Revision: https://phabricator.services.mozilla.com/D122891
Thunderbird can't currently define its own Mach commands, which means
that it doesn't define its own virtualenvs.
However, it does add some tweaks to the `common` virtualenv requirements
(for build scripts, I believe).
So, we should allow it to continue making those tweaks, but we should
remove the capability of specifying PyPI packages. This is needed for
lockfile-generation, since lockfiles are created and placed in the
Mozilla repo, and we don't want "conditional lockfiles" based on the
existence of a "comm" repo.
This will not regress existing Thunderbird expectations.
Differential Revision: https://phabricator.services.mozilla.com/D124515
Rather than deleting the expected target directory of each package
that's being vendored, clear the whole `third_party/python` directory
and re-populate it from scratch.
As part of this, there's an "exclusion" list for packages that can't
be vendored from PyPI.
This has some benefits:
* It'll be harder to forget scraps of files and directories and leave
them in `third_party/python`.
* The exclusion list makes it more clear which packages are managed
manually, and the friction it adds to the workflow will guide
developers to use "requirements.in" instead.
The `test_up_to_date_vendor` test will verify that the vendor directory
is always clean.
Differential Revision: https://phabricator.services.mozilla.com/D123124
Note that, as part of adding this packages to the automated vendoring
system, some dependencies were automatically added - most notably,
dependencies of `taskcluster` that become visible with Python 3.6+.
Also, adds `**/.git` to the exclusions because:
* `.git` is part of our `.hgignore`, but
* `.git` is part of the `aiohttp` `tar.gz` file.
Since the file isn't needed for `pip install`-ing `aiohttp`,
and since we want `./mach vendor python` to be a no-op when there's
no requirement changes, we exclude it.
Differential Revision: https://phabricator.services.mozilla.com/D123122
This required migrating several strings to Fluent, and the bulk of this patch
is those migrations. The rest of the items matched up with an entry in the app
menu, so those items were switched over to use the app menu strings, and one
had to be copied in order to avoid adding a tooltip to the app menu item.
Differential Revision: https://phabricator.services.mozilla.com/D124270
Historically, client.mk was not invoked with -jn because it would create
race conditions, but that was actually mostly solved by the addition of
`.NOTPARALLEL` in bug 422986, although the mechanism of adding -jn via
`MOZ_MAKE_FLAGS` or `MOZ_PARALLEL_BUILD` has continued well past that.
Nowadays, client.mk is only invoked by mach (it will even bail out if
that's not the case) and only has one target (`build`) and no
dependencies.
This means we don't need to rely on `MOZ_PARALLEL_BUILD` to pass `-jn` in
some cases, and can just always invoke `make -f client.mk` with `-jn`, even
when we just want no parallelism, in which case we can use `-j1`.
This, in turn, allows to remove the extra allow_parallel argument to
`_run_make`, and only rely on `num_jobs`, and to remove some of the
multiple ways the `n` in `-jn` could be set.
Differential Revision: https://phabricator.services.mozilla.com/D124729
We had logic in both `mach_bootstrap` and the Mach Bootstrapper to
create the state_dir.
This joins them and has the added benefit of creating the state dir
earlier in the Mach lifecycle (as will be needed for early instantiation
of the Mach virtualenv).
Differential Revision: https://phabricator.services.mozilla.com/D120400
It was originally added to "create-mach-environment" because that would
eventually be called by "./mach bootstrap".
However, as "create-mach-environment" is going away, this moves it
closer to the command that it's meant to impact.
Differential Revision: https://phabricator.services.mozilla.com/D120399
Though it's acceptable for an optional dependency to not be installed,
it *is* a problem if it's installed to the wrong version.
Differential Revision: https://phabricator.services.mozilla.com/D122886
We've overloaded "bootstrap" to mean three different things:
* The "standalone bootstrap script": `python/mozboot/bin/bootstrap.py`.
This is to freshly clone a new repo, then run `./mach bootstrap`.
* `./mach bootstrap`: Install necessary dependencies and set up the
system for development.
* "Mach bootstrap": do the in-process initialization work Mach needs
before it can run commands.
By using the term "initialize" instead, perhaps we can remove
ambiguity when discussing Mach.
I'm not attached to the name (or this change at all), but I'm interested
in reviewer thoughts :)
Differential Revision: https://phabricator.services.mozilla.com/D120410
This required migrating several strings to Fluent, and the bulk of this patch
is those migrations. The rest of the items matched up with an entry in the app
menu, so those items were switched over to use the app menu strings.
Differential Revision: https://phabricator.services.mozilla.com/D124270
There are two problems that this patch addresses:
1) The path to the VS solution file that we're using to launch it is malformed,
because os.path.join is using backslash seperators, but we're passing it a
path which already contains forward slash seperators, and mixing the two is
not valid. This is preventing VS from being launched at all.
2) We're throwing if explorer.exe does not return 0 when we call it to launch
VS, but explorer.exe always returns 1 when run this way, even if it
succeeded, so we output a spurious exception to the console.
Differential Revision: https://phabricator.services.mozilla.com/D124488
Rather than deleting the expected target directory of each package
that's being vendored, clear the whole `third_party/python` directory
and re-populate it from scratch.
As part of this, there's an "exclusion" list for packages that can't
be vendored from PyPI.
This has some benefits:
* It'll be harder to forget scraps of files and directories and leave
them in `third_party/python`.
* The exclusion list makes it more clear which packages are managed
manually, and the friction it adds to the workflow will guide
developers to use "requirements.in" instead.
The `test_up_to_date_vendor` test will verify that the vendor directory
is always clean.
Differential Revision: https://phabricator.services.mozilla.com/D123124
Note that, as part of adding this packages to the automated vendoring
system, some dependencies were automatically added - most notably,
dependencies of `taskcluster` that become visible with Python 3.6+.
Also, adds `**/.git` to the exclusions because:
* `.git` is part of our `.hgignore`, but
* `.git` is part of the `aiohttp` `tar.gz` file.
Since the file isn't needed for `pip install`-ing `aiohttp`,
and since we want `./mach vendor python` to be a no-op when there's
no requirement changes, we exclude it.
Differential Revision: https://phabricator.services.mozilla.com/D123122
In 'gen_test_backend.py' we fallback to an 'EmptyConfig' in the event there is
no build environment. We do this for the benefit of tests that don't need a
Firefox build (like Python unittests). Without it, we wouldn't be able to run
e.g |mach python-test python/mozversioncontrol| unless you had a build.
However since this 'EmptyConfig' doesn't have 'ENABLE_TESTS' set, it means any
manifests defined in a 'moz.build' behind the 'TEST_DIRS' variables will not be
discovered.
Differential Revision: https://phabricator.services.mozilla.com/D120386
Now that `piptools` is no longer installed from the vendored location,
there are no remaining cases where we are doing `install_pip_package()`
with a vendored package.
Differential Revision: https://phabricator.services.mozilla.com/D123241
We've been installing `piptools` so that we can use the `pip-compile`
binary. However, we can just as easily do
`python -m piptools compile ...` instead, which also allows us to
skip a redundant install.
Differential Revision: https://phabricator.services.mozilla.com/D123240
The `tempfile.tempdir` cache value isn't always populated, but
`tempfile.gettempdir()` always is. They both provide the same result.
Differential Revision: https://phabricator.services.mozilla.com/D124261
This adds the sha256 for the ICU4X license as a variable to vendor_rust.py,
so it can be used as we add ICU4X crates. Since there are several projects
ongoing that want to use ICU4X, I think it makes sense to get the license
review done once, early, so that whoever is ready to land code that uses
ICU4X first is not blocked waiting on license review.
Differential Revision: https://phabricator.services.mozilla.com/D123776
Mitchell Hentges
Rob Lemley
Jeff Gilbert
Niklas Baumgardner
Molly Howell
Steve Fink
Jesse Schwartzentruber
Csoregi Natalia
Mike Hommey
Katherine Patenio
Francesco Lodolo (:flod)
Kagami Sascha Rosylight
Sandor Molnar
Cosmin Sabou
Andrew Halberstadt
Noemi Erli
Dan Minor