Граф коммитов

180 Коммитов

Автор SHA1 Сообщение Дата
Mitchell Hentges b3b0d2b63d Bug 1752927: Avoid distutils deprecation warning on every Mach call r=ahal
Python 3.10 prints a warning when it sees `distutils` used, so let's
avoid using `distutils` in the "common Mach code" that runs on every
Mach call.

Note that this change will also fix the `SetuptoolsDeprecationWarning`
that shipped on `setuptools>=58.3.0`, since our usage of `distutils` was
calling a `setup.py install` under the hood.

Differential Revision: https://phabricator.services.mozilla.com/D137499
2022-02-04 21:49:15 +00:00
Mitchell Hentges 16f2e5f5b2 Bug 1751284: Assert for `distutils` during Mach initialization r=ahal
Our platform-specific `bootstrap` code transitively depends on
the standard library's `distutils` module, but Debian Linux and derived
distros generally split `distutils` into a separate installable OS
package from `python3` itself.

So, Python 3 being able to run isn't a sufficient guarantee that
`distutils` is available.

To catch this case, add a `distutils` check at the very beginning of
Mach initialization.

-----

Also remove an obsolete comment claiming that `mach bootstrap` doesn't
need `distutils`, which was obsoleted by Bug 1717051.

Differential Revision: https://phabricator.services.mozilla.com/D136639
2022-01-24 22:35:25 +00:00
Mitchell Hentges 92a19e0eb9 Bug 1740123: Allow invoking Mach outside of MozillaBuild r=glandium
Assuming that the `MOZILLABUILD` environment variable is set, allow
invoking Mach from non-MozillaBuild terminals.

Note that MozillaBuild still needs to be installed, and the
`MOZILLABUILD` environment variable will have to be set.

For future reference: when I tried setting this up with Windows
Store's Python 3.9, I encountered issues when running binaries installed
via `pip`: it would fail with `abort: failed to load Python DLL
python3x.dll`.

Differential Revision: https://phabricator.services.mozilla.com/D133936
2022-01-06 06:49:47 +00:00
Mitchell Hentges 0445e1ce71 Bug 1740123: Move Python version check earlier r=ahal
Move Python version check as early as possible so that more code can
safely depend on modern behaviour while out-of-date Python versions still
get graceful error messages.

Without this change, Python 2 usages fail on importing `importlib.util`
before the nice "out-of-date version" warning is printed.

Differential Revision: https://phabricator.services.mozilla.com/D134185
2022-01-06 00:32:49 +00:00
Mitchell Hentges 7e6a4952a2 Bug 1725895: Port `--profile-command` to pure-Python r=nalexander,glandium
As part of this, the shell-script part of `./mach` can be removed,
making it pure Python.

There's a change in `--profile-command` behaviour, though: it now only
profiles the specific command, rather than all of Mach.
This is because _so much of Mach_ has already been run before
CLI arguments are parsed in the Python process.

If a developer wants to profile Mach itself, they can manually run
`python3 -m cProfile -o <file> ./mach ...`

Differential Revision: https://phabricator.services.mozilla.com/D133928
2022-01-06 00:32:48 +00:00
Marian-Vasile Laza 7262acfd88 Backed out 2 changesets (bug 1725895, bug 1740123) for causing build bustages.
Backed out changeset a6eabd95e31d (bug 1740123)
Backed out changeset bfd5211060b4 (bug 1725895)
2022-01-05 03:46:17 +02:00
Mitchell Hentges 8fff83d33d Bug 1740123: Move Python version check earlier r=ahal
Move Python version check as early as possible so that more code can
safely depend on modern behaviour while out-of-date Python versions still
get graceful error messages.

Without this change, Python 2 usages fail on importing `importlib.util`
before the nice "out-of-date version" warning is printed.

Differential Revision: https://phabricator.services.mozilla.com/D134185
2022-01-04 21:07:32 +00:00
Mitchell Hentges 07f875d674 Bug 1725895: Port `--profile-command` to pure-Python r=nalexander,glandium
As part of this, the shell-script part of `./mach` can be removed,
making it pure Python.

There's a change in `--profile-command` behaviour, though: it now only
profiles the specific command, rather than all of Mach.
This is because _so much of Mach_ has already been run before
CLI arguments are parsed in the Python process.

If a developer wants to profile Mach itself, they can manually run
`python3 -m cProfile -o <file> ./mach ...`

Differential Revision: https://phabricator.services.mozilla.com/D133928
2022-01-04 21:07:32 +00:00
Rob Lemley 3b62e0f6b3 Bug 1731158 - Allow initializing mach from a Thunderbird source repo. r=firefox-build-system-reviewers,mhentges
mach will initialize from comm/build/mach_initialize.py if it's present.
See bug 1731160. The initialization function wraps the one in build/mach_initialize.py,
then extends sys.path and loads more mach_commands.py files

Differential Revision: https://phabricator.services.mozilla.com/D131869
2021-11-29 18:34:58 +00:00
Mitchell Hentges abeedf3bbd Bug 1717051: Automatically create and activate Mach virtualenv r=ahal
Consolidate Mach virtualenv management to the front of the
Mach process. This obsoletes `./mach create-mach-environment`
and simplifies the `sh` portion of the top-level `./mach` script.

This helps ensure that the Mach virtualenv doesn't become
out-of-sync and simplifies the mental model of the Mach
virtualenv situation.

Differential Revision: https://phabricator.services.mozilla.com/D120401
2021-11-24 20:06:33 +00:00
Mitchell Hentges 2f04d64a12 Bug 1717051: Remove "install-moz-phab" from nativecmds r=ahal
It was in `nativecmds` to make finding `pip3` easier.
However, since we're removing the distinction between "regular" and
"native" cmds (Mach will create and activate its venv naturally during
initialization), we need to start pruning the `nativecmds` list.
Fortunately, we can use the stored "external_python" value to decide
where to install "moz-phab" to.

The new behaviour continues to support the following use cases:
* Using Mach with system Python in the `$PATH`.
* Using Mach from within a human-managed virtualenv.
* Either of the above^ in combination with `MACH_USE_SYSTEM_PYTHON`.

Differential Revision: https://phabricator.services.mozilla.com/D120397
2021-11-24 20:06:33 +00:00
Mitchell Hentges f5b0cb5bc1 Bug 1717051: Rename "mach_bootstrap.py" to "mach_initialize.py" r=firefox-build-system-reviewers,glandium
We've overloaded "bootstrap" to mean three different things:
* The "standalone bootstrap script": `python/mozboot/bin/bootstrap.py`.
  This is to freshly clone a new repo, then run `./mach bootstrap`.
* `./mach bootstrap`: Install necessary dependencies and set up the
  system for development.
* "Mach bootstrap": do the in-process initialization work Mach needs
  before it can run commands.

By using the term "initialize" instead, perhaps we can remove
ambiguity when discussing Mach.

I'm not attached to the name (or this change at all), but I'm interested
in reviewer thoughts :)

Differential Revision: https://phabricator.services.mozilla.com/D120410
2021-09-03 20:46:22 +00:00
Mitchell Hentges 1db9ce10c0 Bug 1708260: Remove "imp" usage when loading Mach r=firefox-build-system-reviewers,glandium
We always run on Python 3.6+, so the `imp` usage was dead code.

Differential Revision: https://phabricator.services.mozilla.com/D120398
2021-09-03 20:46:21 +00:00
Nick Alexander cba669901b Bug 1713173 - Add `--profile-command` flag for profiling `mach` commands. r=firefox-build-system-reviewers,mhentges
I chose to do this at the level of the outer Python invocation because:

1. `python -m cProfile ...` handles writing the file and some other
   details.  It's possible to rebuild the functionality -- the tools
   are there -- but the APIs are awkward.

2. this allows to profile `mach` internals, instead of just the
   invoked command's implementation.

This uses the return code of the `get_command` subshell to transmit
the single bit of information "is the flag present".

The Python-level argument is required in order to have `--help` know
about the option and to avoid the `mach` shell script having to filter
arguments.

Differential Revision: https://phabricator.services.mozilla.com/D116151
2021-06-02 21:28:56 +00:00
championshuttler 1a42e404f2 Bug 1700419 - Remove py2commands logic from mach driver.r=ahal
Differential Revision: https://phabricator.services.mozilla.com/D109654
2021-03-26 05:26:47 +00:00
Andrew Halberstadt 2cdfb630ea Bug 1638991 - [mach] Run ./mach telemetry-tests-client with Python 3, r=raphael
Differential Revision: https://phabricator.services.mozilla.com/D109082
2021-03-23 17:24:43 +00:00
Andrew Halberstadt afc041270e Bug 1520458 - Run ./mach raptor with Python 3, r=perftest-reviewers,Bebe,sparky
Differential Revision: https://phabricator.services.mozilla.com/D108828
2021-03-22 14:55:18 +00:00
Andrew Halberstadt e9d6672ac4 Bug 1638992 - Run ./mach test with Python 3, r=releng-reviewers,jmaher
Differential Revision: https://phabricator.services.mozilla.com/D108257
2021-03-12 21:10:31 +00:00
Mitchell Hentges b7e0dfe1d1 Bug 1695272: Move `--no-interactive` to global mach args r=firefox-build-system-reviewers,glandium
Updates `./mach bootstrap` to use `--no-interactive` from global args.
Ensures all bootstrap prompts have a default option.

Differential Revision: https://phabricator.services.mozilla.com/D106814
2021-03-12 16:07:11 +00:00
Andrew Halberstadt 22417e814d Bug 1639024 - Run |mach mozharness| with Python 3, r=releng-reviewers,aki
Differential Revision: https://phabricator.services.mozilla.com/D108111
2021-03-12 14:27:29 +00:00
Andrew Halberstadt 659a890219 Bug 1638970 - Run 'mach jsshell-bench' with Python 3 r=releng-reviewers,mtabara,bhearsum
Differential Revision: https://phabricator.services.mozilla.com/D94882
2021-03-11 17:52:59 +00:00
James Graham caa4eff9e7 Bug 1639003 - Convert mach wpt-metadata-merge to Python 3, r=ahal
This code is generally missing tests, but I tried it out on the wpt-sync
where it's used and with this fix it worked (previously it had broken
because the wpt libraries started to assume Py 3 semantics)

Differential Revision: https://phabricator.services.mozilla.com/D107992
2021-03-11 15:58:51 +00:00
Joel Maher 6bc1ed63b7 Bug 1697048 - run awsy in py3. r=ahal
run awsy in py3

Differential Revision: https://phabricator.services.mozilla.com/D107551
2021-03-09 15:13:46 +00:00
Henrik Skupin 7b8f1c0f27 Bug 1638960 - Run 'mach firefox-ui-functional' with Python 3. r=jgraham
Differential Revision: https://phabricator.services.mozilla.com/D105002
2021-02-13 20:26:14 +00:00
Henrik Skupin d6d841ad6d Bug 1638973 - Run 'mach marionette-test' with Python 3. r=jgraham
Differential Revision: https://phabricator.services.mozilla.com/D105000
2021-02-13 20:26:13 +00:00
Jan de Mooij c692200e2c Bug 1691923 - Don't require Python 2 for mach jstests. r=sfink,gbrown
This fixes a regression from bug 1690784 and is a good idea anyway.

Differential Revision: https://phabricator.services.mozilla.com/D104667
2021-02-11 06:56:43 +00:00
Mike Hommey 2eeafe257c Bug 1687595 - Always get mach modules from subdirectories of where mach is. r=firefox-build-system-reviewers,mhentges
Shortly after mach was introduced, bug 840588 turned mach into a small
wrapper that could be copied in a directory part of $PATH, making that
wrapper load the right thing from the right source directory.

Years later, that setup is not really supported, and it's extra
complexity that can lead to unpleasant surprises.

Differential Revision: https://phabricator.services.mozilla.com/D102376
2021-01-21 01:39:56 +00:00
Mitchell Hentges d1929ccc4a Bug 1686168: Immediately unset __PYVENV_LAUNCHER__ when `mach` starts r=firefox-build-system-reviewers,sheehan,glandium
Catch additional cases where both __PYVENV_LAUNCHER__ is set and we invoke a
virtualenv python by unsetting the variable in a prominent location.
This will specifically catch the `./mach bootstrap` => `./mach artifact` case.

Differential Revision: https://phabricator.services.mozilla.com/D101809
2021-01-20 16:27:09 +00:00
Henrik Skupin 8f891b15aa Bug 1638962 - [geckodriver] Remove broken geckodriver-test mach command. r=webdriver-reviewers,jgraham
Differential Revision: https://phabricator.services.mozilla.com/D100901
2021-01-06 12:36:17 +00:00
James Graham 5f8380513a Bug 1638961 - Run mach geckodriver under Py 3, r=whimboo
Differential Revision: https://phabricator.services.mozilla.com/D100253
2020-12-21 19:08:56 +00:00
Kagami Sascha Rosylight c29f516191 Bug 1639008 - Run wpt-update and wpt-serve on Python 3 r=jgraham,firefox-build-system-reviewers,dmajor
There have been Py3 support patches on WPT. I've used them on Python 3 and haven't seen any unfixed bugs so far.

Differential Revision: https://phabricator.services.mozilla.com/D99292
2020-12-15 14:11:42 +00:00
James Graham fae57b8a1c Bug 1638996 - Run wpt mach commands under Python 3, r=karlcow
Differential Revision: https://phabricator.services.mozilla.com/D99249
2020-12-14 14:11:15 +00:00
Geoff Brown 8a52920aa9 Bug 1638947 - Run 'mach android' with python 3; r=nalexander
Switch 'mach android' to python 3. As far as I can tell, this just works.

Differential Revision: https://phabricator.services.mozilla.com/D96275
2020-11-09 17:54:33 +00:00
Victor Porof 8c1171c7f8 Bug 1638977 - Convert 'mach prettier-format' to run with Python 3, r=sylvestre
Differential Revision: https://phabricator.services.mozilla.com/D93350
2020-11-05 09:45:14 +00:00
Geoff Brown 17cd08802e Bug 1638981 - Run 'mach reftest', 'mach crashtest', and 'mach jsreftest' with python 3; r=bc
These work for me now, thanks to your earlier work on reftest and my earlier
work on mochitest.

Differential Revision: https://phabricator.services.mozilla.com/D94892
2020-10-29 13:51:44 +00:00
Geoff Brown e83b64c0b7 Bug 1638986 - Run 'mach talos-test' with python 3; r=sparky,perftest-reviewers
Differential Revision: https://phabricator.services.mozilla.com/D94770
2020-10-29 00:23:51 +00:00
Cosmin Sabou f654e61a4f Backed out changeset 86e3c90d221f (bug 1638970) for causing Bug 1673914.
CLOSED TREE DONTBUILD
2020-10-28 23:41:26 +02:00
Ben Hearsum 0f48a58046 Bug 1638970: run 'mach jsshell-bench with python 3' r=releng-reviewers,mtabara DONTBUILD
Differential Revision: https://phabricator.services.mozilla.com/D94882
2020-10-28 13:05:49 +00:00
James Graham 2aec04a4c0 Bug 1639006 - Make mach wpt-test-paths run in Python 3, r=karlcow DONTBUILD
Differential Revision: https://phabricator.services.mozilla.com/D92935
2020-10-26 13:14:52 +00:00
James Graham 5af13e9963 Bug 1639002 - Make mach wpt-manifest-update run in Python 3, r=karlcow
Differential Revision: https://phabricator.services.mozilla.com/D92932
2020-10-26 13:14:13 +00:00
James Graham 904eff9577 Bug 1639007 - Make mach wpt-unittest run in Python 3, r=karlcow DONTBUILD
Differential Revision: https://phabricator.services.mozilla.com/D92930
2020-10-26 13:13:29 +00:00
James Graham adcdde0bf9 Bug 1639004 - Make mach wpt-metadata-summary run in Python 3, r=karlcow
Differential Revision: https://phabricator.services.mozilla.com/D92915
2020-10-26 13:12:15 +00:00
Ben Hearsum 427616a2a1 Bug 1598823: remove visualmetrics mach command r=sparky,releng-reviewers,perftest-reviewers,aki DONTBUILD
Differential Revision: https://phabricator.services.mozilla.com/D94160
2020-10-22 15:20:20 +00:00
Ben Hearsum b81bccea51 Bug 1638952: remove check-spidermonkey mach command. r=sfink,releng-reviewers,aki DONTBUILD
Differential Revision: https://phabricator.services.mozilla.com/D94159
2020-10-22 13:12:32 +00:00
Ben Hearsum 6559aa4480 Bug 1638968: run 'mach jsapi-tests' with python 3 r=releng-reviewers,mtabara DONTBUILD
This already seems to work fine with python 3

Differential Revision: https://phabricator.services.mozilla.com/D94336
2020-10-21 17:17:59 +00:00
Ben Hearsum 5d20c1125d Bug 1638994: run 'mach tps-build' with python 3 r=releng-reviewers,aki DONTBUILD
This appears to work fine already with python 3

Differential Revision: https://phabricator.services.mozilla.com/D94311
2020-10-21 16:38:00 +00:00
Ben Hearsum d684a58597 Bug 1638989: run 'mach taskcluster-load-image' with python 3 r=releng-reviewers,aki DONTBUILD
Differential Revision: https://phabricator.services.mozilla.com/D94188
2020-10-21 14:04:50 +00:00
Ben Hearsum 2167d1b284 Bug 1638957: run 'mach devtools-css-db' under python 3. r=releng-reviewers,aki DONTBUILD
It's not 100% clear how long this command will live, but it doesn't look
immediately removable, and it was easy to add python3 support.

Differential Revision: https://phabricator.services.mozilla.com/D94179
2020-10-21 14:04:17 +00:00
Ben Hearsum d28c2049f7 Bug 1638955: run 'mach cramtest' under python 3. r=releng-reviewers,aki DONTBUILD
This command appears to work just fine under python3 already -- I removed it from the py2 list and ran it with various options, and they all worked fine.

Differential Revision: https://phabricator.services.mozilla.com/D94174
2020-10-20 23:39:35 +00:00
Mike Hommey e469210428 Bug 1669874 - Use `command -v` instead of `which` in mach. r=firefox-build-system-reviewers,andi,rstewart
The former is a POSIX shell construct, usually a builtin, and the latter
is an external program that is not guaranteed to be available.

When `which` is not available, a full ./mach build succeeds with this
change.

Differential Revision: https://phabricator.services.mozilla.com/D92864
2020-10-08 14:58:43 +00:00