Minor fix to remoteautomation.py: Increment stdoutlen before any type conversions,
to ensure that it accurately reflects the byte offset in the file.
With this last change, 'mach mochitest' appears to run correctly on Android with Python 3:
switch it over to Python 3.
Differential Revision: https://phabricator.services.mozilla.com/D91586
Since `install-moz-phab` is meant to simplify the moz-phab setup flow,
automatically prompting for Phabricator credentials removes an otherwise
manual step.
Detecting the "console_script" location of a package in a
cross-platform, virtualenv-supporting and "--user"-supporting way is
tough, and the most consistent solution seems to be to list the package
contents of moz-phab and look for the one that seems to be the entry
point.
Differential Revision: https://phabricator.services.mozilla.com/D90642
This is only useful for `mach` commands that we want to run with Python 3 by default, but for which running with Python 2 is still useful. We now have one such command: `python-test`.
In `mach`, switch on the presence of the `MACH_PY2` environment variable. We only want to allow this for `python-test`, so do that sanity check in `mach` as well.
Differential Revision: https://phabricator.services.mozilla.com/D89162
In two different places we've been encountering issues regarding 1) how we configure the system Python environment and 2) how the system Python environment relates to the `virtualenv`s that we use for building, testing, and other dev tasks. Specifically:
1. With the push to use `glean` for telemetry in `mach`, we are requiring (or rather, strongly encouraging) the `glean_sdk` Python package to be installed with bug 1651424. `mach bootstrap` upgrades the library using your system Python 3 in bug 1654607. We can't vendor it due to the package containing native code. Since we generally vendor all code required for `mach` to function, requiring that the system Python be configured with a certain version of `glean` is an unfortunate change.
2. The build uses the vendored `glean_parser` for a number of build tasks. Since the vendored `glean_parser` conflicts with the globally-installed `glean_sdk` package, we had to add special ad-hoc handling to allow us to circumvent this conflict in bug 1655781.
3. We begin to rely more and more on the `zstandard` package during build tasks, this package again being one that we can't vendor due to containing native code. Bug 1654994 contained more ad-hoc code which subprocesses out from the build system's `virtualenv` to the SYSTEM `python3` binary, assuming that the system `python3` has `zstandard` installed.
As we rely more on `glean_sdk`, `zstandard`, and other packages that are not vendorable, we need to settle on a standard model for how `mach`, the build process, and other `mach` commands that may make their own `virtualenv`s work in the presence of unvendorable packages.
With that in mind, this patch does all the following:
1. Separate out the `mach` `virtualenv_packages` from the in-build `virtualenv_packages`. Refactor the common stuff into `common_virtualenv_packages.txt`. Add functionality to the `virtualenv_packages` manifest parsing to allow the build `virtualenv` to "inherit" from the parent by pointing to the parent's `site-packages`. The `in-virtualenv` feature from bug 1655781 is no longer necessary, so delete it.
2. Add code to `bootstrap`, as well as a new `mach` command `create-mach-environment` to create `virtualenv`s in `~/.mozbuild`.
3. Add code to `mach` to dispatch either to the in-`~/.mozbuild` `virtualenv`s (or to the system Python 3 for commands which cannot run in the `virtualenv`s, namely `bootstrap` and `create-mach-environment`).
4. Remove the "add global argument" feature from `mach`. It isn't used and conflicts with (3).
5. Remove the `--print-command` feature from `mach` which is obsoleted by these changes.
This has the effect of allowing us to install packages that cannot be vendored into a "common" place (namely the global `~/.mozbuild` `virtualenv`s) and use those from the build without requiring us to hit the network. Miscellaneous implementation notes:
1. We allow users to force running `mach` with the system Python if they like. For now it doesn't make any sense to require 100% of people to create these `virtualenv`s when they're allowed to continue on with the old behavior if they like. We also skip this in CI.
2. We needed to duplicate the global-argument logic into the `mach` script to allow for the dispatch behavior. This is something we avoided with the Python 2 -> Python 3 migration with the `--print-command` feature, justifying its use by saying it was only temporarily required until all `mach` commands were running with Python 3. With this change, we'll need to be able to determine the `mach` command from the shell script for the forseeable future, and committing to this forever with the cost that `--print-command` incurs (namely `mach` startup time, an additional .4s on my machine) didn't seem worth it to me. It's not a ton of duplicated code.
Differential Revision: https://phabricator.services.mozilla.com/D85916
|./mach try| subcommands are now compatible with both python 2 and 3.
Hand-tested with many combinations of subcommand and subcommand flags.
Updates tryselect unit tests to use Python 3.
Differential Revision: https://phabricator.services.mozilla.com/D73398
It's not maintained and doesn't work as-is. If we want to revive it we can grab it from source control later, but in the meantime it's just confusing and it comes up in search results even though we never look for it.
Also delete `mach analyze` which depends on the existence of a Tup backend.
Differential Revision: https://phabricator.services.mozilla.com/D73911
It's not maintained and doesn't work as-is. If we want to revive it we can grab it from source control later, but in the meantime it's just confusing and it comes up in search results even though we never look for it.
Also delete `mach analyze` which depends on the existence of a Tup backend.
Differential Revision: https://phabricator.services.mozilla.com/D73911
This uses the `email` module to parse the .patch file that Github generates,
so that it properly decodes encoded-words in the headers. Also using this
module is better with python3, so this patch also takes the command off the
python2 whitelist and makes it python3-compatible.
Differential Revision: https://phabricator.services.mozilla.com/D66621
--HG--
extra : moz-landing-system : lando
This fixes all of the jit-test problems I encountered, and fixes all but the wpt portion of jstests. This is unnecessary, but it also ports jstests from optparse to argparse (because optparse is deprecated, because I like argparse better, and because it happened to fix the minor compatibility issue I ran into even though it has a trivial fix on its own.)
Differential Revision: https://phabricator.services.mozilla.com/D65047
--HG--
extra : moz-landing-system : lando
This command takes a list of test ids and returns the list of test
files that correspond to those test ids. The mapping is one-to-many
between files and test ids i.e. a single file can generate many tests.
Differential Revision: https://phabricator.services.mozilla.com/D56192
--HG--
extra : moz-landing-system : lando
Changes:
Remove `environment` from the python3 blacklist.
Remove the option to specify `configure` for the formatting argument, since it has never been supported (the method has never been implemented).
Differential Revision: https://phabricator.services.mozilla.com/D54395
--HG--
extra : moz-landing-system : lando
This allows running the jit_test.py harness using mach. It works very
similarly to the existing 'mach jstests' command.
Differential Revision: https://phabricator.services.mozilla.com/D51518
--HG--
extra : moz-landing-system : lando
Changes:
- remove `uuid` from the python3 blacklist in `mach`
- enable `test_telemetry.py` for python3
- adjust test outcome expectation for Windows + python3, suspect `mozpack.path` is not filtering Windows path correctly in python3 environment
- switch file read mode between `r` and `rb` depending on version of python
Differential Revision: https://phabricator.services.mozilla.com/D45903
--HG--
extra : moz-landing-system : lando
Some commands use external argument parsers, so invoking |mach help <command>| will import
external modules (which may only be Python 2 compatible).
This makes sure that we detect the actual subcommand we're generating help for
and use the proper Python.
A much simpler solution would have been to run |mach help| with Python 2 all
the time. However, as we convert things to Python 3 this would have meant that
Python 3 only code would blow up. This would have forced us to continue
supporting Python 2, even for Python 3-only commands.
Differential Revision: https://phabricator.services.mozilla.com/D43989
--HG--
extra : moz-landing-system : lando
Changes:
- remove binary encoding when the subdirectories are being collected
- do not open file in binary mode when reading from `telemetry_client_id.json`
Differential Revision: https://phabricator.services.mozilla.com/D44057
--HG--
extra : moz-landing-system : lando
This includes the following commands:
* dxr
* google
* mdn
* search
* searchfox
Differential Revision: https://phabricator.services.mozilla.com/D36843
--HG--
extra : moz-landing-system : lando
Credit: mars for making the shell POSIX compliant
This embeds a blacklist of every mach command that needs to run with Python 2
directly in the mach driver itself. Initially this is every mach command. We
then use a bit of shell to determine whether the command being run needs Python
2 or 3.
While this approach may seem a bit hacky, it has several benefits:
1. No need to add complex machinery in mach's registration code.
2. No need to spawn two separate Python interpreters in the event a different
Python from the original interpreter is needed.
3. Perf impact is negligible.
4. New commands are Python 3 by default.
It is also only a temporary hack. Once all commands are running with Python 3,
we can revert back to the original mach driver.
Differential Revision: https://phabricator.services.mozilla.com/D36103
--HG--
extra : moz-landing-system : lando
This generalizes the monkey-patching of the main module added
in bug 914563 to allow multiprocessing to be used from config.status
(which triggers the same bug as when it's used with `mach` as main).
MozReview-Commit-ID: AdOdpKzmbsp
The mach binary needs to find a srcdir (or test zip) to operator on. As a last resort,
it checks the directory containing the binary itself. If the binary has been symlinked
to say, $HOME/bin, it won't follow the symlink to the proper location, and the check
will fail.
Fixing this means developers can run `mach` from outside of the srcdir, as the symlink
will be followed to the actual srcdir. The motivation for this is actually to improve
the UX of debugging failures on interactive workers. With this fix, we can provide
developers with a mach environment directly from $HOME without them needing to go
find it.
MozReview-Commit-ID: ZEH8MuyKv
--HG--
extra : rebase_source : 2b5adbad1b5261f2268f6a30e7dd884d7820da5e
Currently the mach binary will look for 'mozinfo.json' and if it finds it assume
we are in an objdir. But now, 'mozinfo.json' can also be found in an extracted
tests.zip (depending which zips were used). This trips up the tests.zip mach
environment and causes some problems.
This works around the issue by looking for both 'mozinfo.json' and 'config.status'
MozReview-Commit-ID: BIXcCm4LzE2
--HG--
extra : rebase_source : 6e2dc0502de1cddec8eccdfe30524864be81536c
When writing bytecode, Python will append "c" to the loaded filename to
produce a bytecode file. Since "mach" was being imported, this resulted
in the creation of a "machc" file.
The implementation of imp.load_module() in CPython's import.c checks
sys.dont_write_bytecode. So, we wrap imp.load_module to set this flag
when importing mach.
--HG--
extra : rebase_source : 248a2349663affee3920a0726e10818d57c6ff17
extra : amend_source : 221280da9963cf91975658144ff3011353852fee
Geoff Brown
Mitchell Hentges
Steve Fink
Ricky Stewart
Hamzah
Tom Prince
Geoff Brown
Edgar Chen
Mihai Alexandru Michis
egao
Andi-Bogdan Postelnicu
Noemi Erli
Bob Clary
James Graham
Nicholas Nethercote
Dan Minor
Henrik Skupin
Simon Fraser
Razvan Maries
Mike Hommey
Stefan Hindli
Coroiu Cristina
William Lachance
Kartikaya Gupta
Bogdan Tara
Divya Rani
Axel Hecht
Sylvestre Ledru
Kris Maglione
Ted Campbell
Andrew Halberstadt
Oana Pop Rus
championshuttler
Chris Manchester
Ehsan Akhgari
Gregory Szorc