Use `poetry` instead of `pip-compile`, then leverage `pip` to assert
that there's no conflicts between environments.
This enables simplifying `requirements.in`, since `colorama` is now
automatically picked up, as vendoring is no longer linux-py3.6-specific.
`poetry==1.2.0a2` was chosen because it was the first version that would
include `pip`/`setuptools`/`wheel` in the `requirements.txt`, yet it was
also the last version that supports Python 3.6 (as still used in CI in
some spots).
Sorry about the `requirements.txt` noise, that's because poetry doesn't
sort hashes yet [1]. Fortunately, you can be confident in
`requirements.txt` changes by checking that this patch doesn't include
changes to vendored packages themselves (I'm including the results of a
`./mach vendor python`).
[1] https://github.com/python-poetry/poetry/issues/5408
Differential Revision: https://phabricator.services.mozilla.com/D142884
`coverage` has native code, so the vendored version was only used as
source code from which the actual package could be built.
Since its always used in a context where we can `pip install` over the
network, let's do that. This cleans up our tree a bit and allows us to
leverage the `coverage` wheels.
Differential Revision: https://phabricator.services.mozilla.com/D138816
This picks up a change that uses the repo root (when it is able to be detected)
to populate the default parameters. This is needed because there are some Gecko
invocations of `mach taskgraph` that happen outside of the srcdir, so we can't
rely on cwd to find the repository.
Differential Revision: https://phabricator.services.mozilla.com/D136517
A bunch of modern packages (`pytest`, `twisted`, `automat`) all need
`attrs==19.2.0` (or newer).
We _could_ bump `attrs` all the way to the modern `21.4.0` version, but
I'd like to defer that upgrade risk, since there's a
lot of backwards-incompatible changes and deprecations. So, lightly bump
it to `19.2.0`.
As part of bumping it, `pytest` is no longer compatible.
The earliest candidate that seems to be compatible is `pytest` 4.6.6,
which boasts in its release notes that it's resolved some deprecation
warnings against `attrs>=19.2.0`.
Once `pytest` was bumped, it needed a newer version of `pluggy`, which
itself has dependencies.
Since we're using hashes in `tox_requirements.txt`, all dependencies
needed to be hashed as well.
Differential Revision: https://phabricator.services.mozilla.com/D135178
A bunch of modern packages (`pytest`, `twisted`, `automat`) all need `attrs==19.2.0` (or newer).
We _could_ bump `attrs` all the way to the modern `21.4.0` version, but I'd like to defer that upgrade risk, since there's a
lot of backwards-incompatible changes and deprecations. So, lightly bump it to `19.2.0`.
Differential Revision: https://phabricator.services.mozilla.com/D135178
This is needed to pick up new functionality around setting parameter defaults
via `extend_parameter_schema`.
Depends on D134514
Differential Revision: https://phabricator.services.mozilla.com/D134967
Also updates the docs on how to update the glean_parser in-tree.
Also adds a `no_lint` exception to test pings to avoid breaking the
build.
Differential Revision: https://phabricator.services.mozilla.com/D133077
Also updates the docs on how to update the glean_parser in-tree.
Also adds a `no_lint` exception to test pings to avoid breaking the
build.
Differential Revision: https://phabricator.services.mozilla.com/D133077
This is needed because `black==21.11b1` requires `click>=7.1.2`
`pathspec<1,>=0.9.0`, and `typing-extensions>=3.10.0.0`.
Differential Revision: https://phabricator.services.mozilla.com/D127163
This is needed because `black==21.11b1` requires `click>=7.1.2`
`pathspec<1,>=0.9.0`, and `typing-extensions>=3.10.0.0`.
Differential Revision: https://phabricator.services.mozilla.com/D127163
The virutalenv hack is in the fedora-distributed version of
virtualenv... Presumably eventually will become unnecessary once they
provide a proper "venv" distutils?
This patch applies both before and after the bump in comment 5, so your
call.
Differential Revision: https://phabricator.services.mozilla.com/D130410
`tqdm` allows rendering and updating a single-line progress bar, which
is useful for all sorts of different work.
Our first use case for this will be replacing `wget` with a pure-python
downloader.
Source here: https://github.com/tqdm/tqdm
Differential Revision: https://phabricator.services.mozilla.com/D129094
Now that are prioritizing system over virtualenv site-packages, the
system `pip` is sometimes being used instead.
This is causing issues when the system pip is set up in a
distro-specific way, such as when "debundled":
https://github.com/pypa/pip/blob/9.0.1/pip/_vendor/__init__.py#L53-L61
However, if we vendor `pip`, `setuptools` and `wheel`, and ensure that
they're prioritized in the `sys.path` before anything is imported from
the system, then we can ensure that we're using a modern `pip` _and_
sidestep system-specific pip weirdness.
Note that `pip-compile`'s `--allow-unsafe` flag is not as dangerous as
it sounds.
There's confusion among maintainers about its origin:
https://github.com/jazzband/pip-tools/issues/522
Additionally, it's going to be enabled by default in a future
`pip-tools` release. So, it's not scary for us to embrace here.
Also, heads up that the "pip outdated warning" no longer needs
to be manually silenced, since pip avoids that code path when
not running from an "installed" context.
Differential Revision: https://phabricator.services.mozilla.com/D127182
I'd like to vendor the 'taskcluster-taskgraph' pacakge, but it requires slugid
>= 2.0. In order to update that we also need to update 'taskcluster' and
'taskcluster_urls' as the currently vendored versions don't work with
'slugid>=2.0' (due to slugid returning str rather than bytes here).
Version changes are:
* taskcluster 6.0.0 -> 44.2.2
* taskcluster_urls 11.0.0 -> 13.0.1
* slugid 1.0.7 -> 2.0.0
Differential Revision: https://phabricator.services.mozilla.com/D127680
* `pystache` is unused.
* `funcsigs` is only needed for WPT - use the WPT version instead.
* `moz.build` has a bunch of obsolete/redundant references, clean
them up.
* `mohawk` isn't used directly, but is rather depended-on via
`taskcluster`. So, remove it from `requirements.in`.
Differential Revision: https://phabricator.services.mozilla.com/D126732
`pytest` has platform-specific dependencies (`colorama` on Windows), but
our vendoring logic currently doesn't support having platform-specific
packages.
Since `python-test` jobs don't have to be isolated from the network,
migrate `pytest` to be a `pypi:` dependency.
Differential Revision: https://phabricator.services.mozilla.com/D126285
This will allow us to parse and compare pip package versions the same
way that `pip` does.
`pyparsing` was added because it's needed by `packaging`.
Differential Revision: https://phabricator.services.mozilla.com/D122888
The `*.egg-info` directories are needed for the packages to show up as
"distributions" to `pip` and other environment-checking logic.
We know that `*.egg-info` directories are cross-platform because they
exist in the globally-usable `tar.gz` releases of packages.
Differential Revision: https://phabricator.services.mozilla.com/D125909
This will allow us to parse and compare pip package versions the same
way that `pip` does.
`pyparsing` was added because it's needed by `packaging`.
Differential Revision: https://phabricator.services.mozilla.com/D122888
The `*.egg-info` directories are needed for the packages to show up as
"distributions" to `pip` and other environment-checking logic.
We know that `*.egg-info` directories are cross-platform because they
exist in the globally-usable `tar.gz` releases of packages.
Differential Revision: https://phabricator.services.mozilla.com/D125909
Rather than deleting the expected target directory of each package
that's being vendored, clear the whole `third_party/python` directory
and re-populate it from scratch.
As part of this, there's an "exclusion" list for packages that can't
be vendored from PyPI.
This has some benefits:
* It'll be harder to forget scraps of files and directories and leave
them in `third_party/python`.
* The exclusion list makes it more clear which packages are managed
manually, and the friction it adds to the workflow will guide
developers to use "requirements.in" instead.
The `test_up_to_date_vendor` test will verify that the vendor directory
is always clean.
Differential Revision: https://phabricator.services.mozilla.com/D123124
Note that, as part of adding this packages to the automated vendoring
system, some dependencies were automatically added - most notably,
dependencies of `taskcluster` that become visible with Python 3.6+.
Also, adds `**/.git` to the exclusions because:
* `.git` is part of our `.hgignore`, but
* `.git` is part of the `aiohttp` `tar.gz` file.
Since the file isn't needed for `pip install`-ing `aiohttp`,
and since we want `./mach vendor python` to be a no-op when there's
no requirement changes, we exclude it.
Differential Revision: https://phabricator.services.mozilla.com/D123122
Note that this patch makes modifications to the vendored
`virtualenv` package by removing the modern `setuptools`
packages, replacing them with `51.2.0`. This is because
`51.3.0` somehow causes xpcshell failures with the following
Python bug: https://bugs.python.org/issue37380
This upgrades:
* `pip` 20.3.1 => 21.2.3
* `setuptools` 51.0.0 => 51.2.0
* `wheel` 0.36.1 => 0.37.0
Differential Revision: https://phabricator.services.mozilla.com/D123120
Andrew Halberstadt
Mike Hommey
Butkovits Atila
Mitchell Hentges
Kelsey Gilbert
Thomas Wisniewski
Jan-Erik Rediger
Marian-Vasile Laza
Iulian Moraru
Narcis Beleuzu
Emilio Cobos Álvarez
Alex Hochheiden
criss