`mozversioncontrol` runs currently use `universal_newlines`, which
is an alias for `text`. This forces the command output file object
to be opened in text mode using the platform's default encoding. On
Windows the default encoding is usually a single-byte encoding such
as `cp-1252`, which can cause issues when parsing patches that include
multi-byte sequences. Add an `encoding` option to `_run` with `utf-8`
as the default option and change the `check_output` call to use it.
Callers can revert to the previous behaviour quite easily by setting
`encoding=None` if necessary.
Differential Revision: https://phabricator.services.mozilla.com/D190710
Add support for submitting stacks of commits to Lando for queueing
on Try, instead of pushing to hg.mozilla.org directly. This patch
implements the Device Code Authorization flow for Auth0, simple changeset
discovery and patch gathering, and submissing to Lando via HTTP POST.
Add a `try.txt` virtualenv site that contains packages from the common
virtualenv as well as the `auth0-python` package for verifying Auth0
JWTs. Use this new virtualenv for `mach try` and related subcommands.
Add a `--push-to-lando` flag that controls whether the push will be made
via the Lando API or using the VCS to hg.mozilla.org directly.
Create a `lando.py` module in the `tryselect` package that handles the
details around submitting to Lando. Authentication is handled by the
Device Code Authorization flow, and the returned access token is saved
to the mozbuild state directory. Auth0 details are added to the `.lando.ini`
file in the repo root, and a `LANDO_TRY_USE_DEV` environment variable can
be set to control submitting to the prod or dev Lando environments.
This module also includes patch stack gathering and discovery via `mozversioncontrol`.
mozversioncontrol's `Repository` subclass is extended with helper functions
to gather patch files from Mercurial and Git. We also add a `try_config_commit`
context manager that creates a temporary commit to hold try syntax commit
messages and/or a `try_task_config.json` file, which is removed from version
control on submission.
The `mach try` cram tests now use a separate virtualenv that must be built
before running, causing unexpected output in the tests. Run `mach try --help`
in the test setup to force the virtualenv to be built before running any
test.
`mach try chooser` would previously install packages for a small web application
from a `requirements.txt` file when run. Install these via the new `try.txt`
site environment instead. The required dependencies are also added to the
`python-test.txt` site since they will cause some tests to fail due to the
missing dependencies.
Differential Revision: https://phabricator.services.mozilla.com/D187641
This results in some changes from our current `isort` configuration. I'm
unclear if it's because ruff isn't at 100% parity with isort, they choose
different defaults or if I missed some configuration.
Either way, the changes all look reasonable to me (see child commit), so I'm
inclined to just accept the new import format it imposes.
Differential Revision: https://phabricator.services.mozilla.com/D172348
This results in some changes from our current `isort` configuration. I'm
unclear if it's because ruff isn't at 100% parity with isort, they choose
different defaults or if I missed some configuration.
Either way, the changes all look reasonable to me (see child commit), so I'm
inclined to just accept the new import format it imposes.
Differential Revision: https://phabricator.services.mozilla.com/D172348
This patch adds a `capture_log` flag that will allow a user to capture the logs produced when running the `push_to_try` methods. Currently, we use `subprocess.check_call` which causes hg to hang when we attempt to redirect and gather stdout. Using `subprocess.run` is better as it lets us capture the logs, but the log output is very slow and can easily lead people to believe that hg is hanging when it's not. This results in corrupted repos. Using Popen, the logs are output slower than `check_call`, but faster than `run` so you know something is happening. The speed at which these logs are printed is also why I have this log capturing behind a flag.
This functionality will be used in the `./mach try perf` selector to capture the child revisions produced for the try task config file changes and allow us to produce a PerfCompare link to provide the user.
Differential Revision: https://phabricator.services.mozilla.com/D155980
Following the pytest "deprecations and removals" docs [1], this patch:
* Replaces `@pytest.yield_fixture` with `@pytest.fixture`.
* Replaces `.funcargnames` with `.fixturenames`.
* Uses `Pathlib` parameter instead of the `py.path.local` one in
associated `pytest_*` hooks.
* Replaces `--strict` with `--strict-markers`
[1] https://docs.pytest.org/en/latest/deprecations.html
Differential Revision: https://phabricator.services.mozilla.com/D138815
Though not a panacea, this patch addresses existing usages of
`get_repository_from_build_config()` to fall back to
`get_repository_object()` if `MissingVCSTool` was encountered.
Ideally, we'd update `substs["HG"]` when we detect that it's changed,
but we don't have infrastructure to incrementally change configure
output.
So, other issues may creep up that will still warrant a fresh
`./mach configure`.
Differential Revision: https://phabricator.services.mozilla.com/D137675
The py.path allowed adding arbitrary fields to it, while a pathlib.Path
does not. To replicate the behavior with as few changes as possible, I
made a test fixture wrapper class (RepoTestFixture) to add the same
arbitrary fields to, while now containing a Pathlib path instead.
Also added a utility functions to make the running of tests a
bit cleaner ('execute_next_step')
Differential Revision: https://phabricator.services.mozilla.com/D136302
Note: In `test_commit.py` For date formatting, the `%s` did not work on Windows.
On Linux it gives the time in seconds. The switch to the rfc822date format for
both hg and git `%aD` made the expected output in the check later work for both
git and hg, and made it platform agnostic.
Differential Revision: https://phabricator.services.mozilla.com/D136209
If watchman is invoked (as part of hg's fsmonitor) with a path whose case does
not match the on-disk case (e.g., for case-insensitive file systems like NTFS),
it will get angry and throw a RootResolveError (although it will still work).
When doing `./mach try`, we end up invoking watchman several times which leads
to it getting very angry and throwing lots of errors.
This is all due to how we compute virtualenv paths -- by normcase()-ing them in
`mach.pth` -- which results in `push.py` having a lowercase path *even if* your
mozilla-central checkout has a non-lowercase on-disk path.
Calling Path.resolve() on the resulting path returns it to its on-disk
representation and no longer angers watchman.
Differential Revision: https://phabricator.services.mozilla.com/D133397
Unless an upstream is specified, we should be able to detect which files
are changed on our local branch without hitting the network.
Differential Revision: https://phabricator.services.mozilla.com/D130684
To track whether issues being reported are for old versions, we attach
an hg revision hash as a release version with each report.
We use the base_ref because:
* It's a public revision that we can look up, and
* Since we only submit Sentry events when mach is unmodified,
it's essentially acting as a version identifier.
Differential Revision: https://phabricator.services.mozilla.com/D108307
Allow-list all Python code in tree for use with the black linter, and re-format all code in-tree accordingly.
To produce this patch I did all of the following:
1. Make changes to tools/lint/black.yml to remove include: stanza and update list of source extensions.
2. Run ./mach lint --linter black --fix
3. Make some ad-hoc manual updates to python/mozbuild/mozbuild/test/configure/test_configure.py -- it has some hard-coded line numbers that the reformat breaks.
4. Make some ad-hoc manual updates to `testing/marionette/client/setup.py`, `testing/marionette/harness/setup.py`, and `testing/firefox-ui/harness/setup.py`, which have hard-coded regexes that break after the reformat.
5. Add a set of exclusions to black.yml. These will be deleted in a follow-up bug (1672023).
# ignore-this-changeset
Differential Revision: https://phabricator.services.mozilla.com/D94045
Allow-list all Python code in tree for use with the black linter, and re-format all code in-tree accordingly.
To produce this patch I did all of the following:
1. Make changes to tools/lint/black.yml to remove include: stanza and update list of source extensions.
2. Run ./mach lint --linter black --fix
3. Make some ad-hoc manual updates to python/mozbuild/mozbuild/test/configure/test_configure.py -- it has some hard-coded line numbers that the reformat breaks.
4. Make some ad-hoc manual updates to `testing/marionette/client/setup.py`, `testing/marionette/harness/setup.py`, and `testing/firefox-ui/harness/setup.py`, which have hard-coded regexes that break after the reformat.
5. Add a set of exclusions to black.yml. These will be deleted in a follow-up bug (1672023).
# ignore-this-changeset
Differential Revision: https://phabricator.services.mozilla.com/D94045
Allow-list all Python code in tree for use with the black linter, and re-format all code in-tree accordingly.
To produce this patch I did all of the following:
1. Make changes to tools/lint/black.yml to remove include: stanza and update list of source extensions.
2. Run ./mach lint --linter black --fix
3. Make some ad-hoc manual updates to python/mozbuild/mozbuild/test/configure/test_configure.py -- it has some hard-coded line numbers that the reformat breaks.
4. Add a set of exclusions to black.yml. These will be deleted in a follow-up bug (1672023).
# ignore-this-changeset
Differential Revision: https://phabricator.services.mozilla.com/D94045
If the "ui.username" config option doesn't have a value in the expected
format, assume that the user email address is unknown instead of
throwing an exception.
Differential Revision: https://phabricator.services.mozilla.com/D91626
These arguments will get converted to `bytes` later on if necessary; if we don't defer to `hglib`, then we need proper strings.
Differential Revision: https://phabricator.services.mozilla.com/D87440
This resolves a long-standing issue in development where `mach artifact` (and therefore `mach bootstrap`) would fail unpredictably if you had dirty, but ignored, files in your checkout. Resolving this problem often required unwieldy `hg purge`/`git ignore` incantations that are easy to get wrong.
This patch addresses the problem by doing what we "should" have been doing all along, and consulting the VCS to list tracked files rather than listing EVERY file on disk and applying heuristics to determine whether they should be included in the hash.
Differential Revision: https://phabricator.services.mozilla.com/D86780
Without this patch, the last "path" in this list will always be the empty string due to how the `-z` option to `git` works. This mirrors what we already do in the `get_files_in_working_directory` implementation for `hg`.
Differential Revision: https://phabricator.services.mozilla.com/D86752
Connor Sheehan
Steve Fink
Sandor Molnar
ahochheiden
andrej
nicklas boman
Cosmin Sabou
Iulian Moraru
Andrew Halberstadt
Cristian Tuns
Ho Cheung
Marco Castelluccio
Greg Mierzwinski
Mitchell Hentges
Makoto Kato
Barret Rennie
surajeet310
Andrew Halberstadt
Ben Hearsum
Butkovits Atila
Ricky Stewart
Cristina Coroiu
Bogdan Tara
Dorel Luca