Changes:
- remove `uuid` from the python3 blacklist in `mach`
- enable `test_telemetry.py` for python3
- adjust test outcome expectation for Windows + python3, suspect `mozpack.path` is not filtering Windows path correctly in python3 environment
- switch file read mode between `r` and `rb` depending on version of python
Differential Revision: https://phabricator.services.mozilla.com/D45903
--HG--
extra : moz-landing-system : lando
This logic is very 'mozill-central' specific and should live outside of mach
core if possible. Luckily we already have a concept of a 'pre_dispatch_handler'
that is meant for exactly this type of use case.
Differential Revision: https://phabricator.services.mozilla.com/D47668
--HG--
extra : moz-landing-system : lando
The mach driver will now run all misspelled commands with Python 3. That means
we can't automatically execute the suggested command anymore, as it may need to
run against Python 2.
Ideally we could figure out a way to check the command against the 'mach'
whitelist, but until then, let's just disable automatic execution. Worst case
scenario we can turn it back on after the migration has finished.
Differential Revision: https://phabricator.services.mozilla.com/D44282
--HG--
extra : moz-landing-system : lando
Some commands use external argument parsers, so invoking |mach help <command>| will import
external modules (which may only be Python 2 compatible).
This makes sure that we detect the actual subcommand we're generating help for
and use the proper Python.
A much simpler solution would have been to run |mach help| with Python 2 all
the time. However, as we convert things to Python 3 this would have meant that
Python 3 only code would blow up. This would have forced us to continue
supporting Python 2, even for Python 3-only commands.
Differential Revision: https://phabricator.services.mozilla.com/D43989
--HG--
extra : moz-landing-system : lando
Credit: mars for making the shell POSIX compliant
This embeds a blacklist of every mach command that needs to run with Python 2
directly in the mach driver itself. Initially this is every mach command. We
then use a bit of shell to determine whether the command being run needs Python
2 or 3.
While this approach may seem a bit hacky, it has several benefits:
1. No need to add complex machinery in mach's registration code.
2. No need to spawn two separate Python interpreters in the event a different
Python from the original interpreter is needed.
3. Perf impact is negligible.
4. New commands are Python 3 by default.
It is also only a temporary hack. Once all commands are running with Python 3,
we can revert back to the original mach driver.
Differential Revision: https://phabricator.services.mozilla.com/D36103
--HG--
extra : moz-landing-system : lando
On Windows in Python 2, the subprocess module requires the use of bytes with
the 'env' argument. For that reason, we would sometimes use byte strings with
'os.environ' like so:
os.environ[b"FOO"] = b"bar"
However, this is a failure with Python 3 as 'os.environ' must only be used with
the text type. This patch creates a new 'setenv' helper that ensures we create
new environment with 'bytes' on Python 2, and 'text' on Python 3.
Differential Revision: https://phabricator.services.mozilla.com/D38237
--HG--
extra : moz-landing-system : lando
This test is working with minimal effort. Let's get it running to prevent
future regressions.
Differential Revision: https://phabricator.services.mozilla.com/D36098
--HG--
extra : moz-landing-system : lando
We want `mach run` for Android to be wildly different than `mach run`
for Desktop. But right now, mach really doesn't support two different
implementations of the same underlying named command. The avenues
that might support different implementations, mostly run through
`conditions`.
`conditions` were added to mach commands in Bug 901972, and never
really anticipated this use case: commands are keyed by name condition
evaluation is delayed until dispatch-time. In order to have different
commands with the same name, and have full support for `--help`,
command matching, suggestions, etc, we really need conditions to
evaluate at parse-time. Indeed, since Bug 901972 landed, we've moved
context creation earlier in the dispatch flow and hacked in things
that look like parse-time conditions (see Bug 1291335 and Bug
1305695).
This approach is not the prettiest, but it handles this narrow
use-case -- making `mach run` and `mach install` different on Android
-- without much code churn.
Differential Revision: https://phabricator.services.mozilla.com/D18290
--HG--
extra : moz-landing-system : lando
For mach commands that have 'pass_context=True', we should implicitly add the
handler instance to the context. This will give mach command implementations an
easy way to access things like the command/subcommand names, the parser, argv
list, etc.
Differential Revision: https://phabricator.services.mozilla.com/D20521
--HG--
extra : moz-landing-system : lando
This fixes an issue from bug 1518586.
It stemmed from the fact that I misunderstood how the 'parser' attribute
was being used in mach commands and didn't do enough testing. This patch
fixes things such that we check that 'parser' is not None, as well as
add the completion targets from 'handler.arguments'.
Differential Revision: https://phabricator.services.mozilla.com/D16501
--HG--
extra : moz-landing-system : lando
Since we're calling into a mach command for the current completion
implementation anyway (and incurring python startup penalties), we
might as well move all the bash logic into the mach command.
The new 'mach-completion' command was created in case there are
scripts relying on the current behaviour of 'mach-commands'.
Depends on D16254
Differential Revision: https://phabricator.services.mozilla.com/D16255
--HG--
extra : moz-landing-system : lando
I was looking for a good place to put documentation for enabling the
bash completion script when I realized that the 'mach' documentation
is targeted at tool developers rather than users.
Seeing as this is the main 'mach' documentation and we want to make
'firefox-source-docs' the place to go for contribution information,
we should endeavour to target end users with this documentation.
This adds a very basic usage page which should be expanded upon, but
is better than nothing.
I changed the headers in 'index.rst' to use raw:: html so that they
don't show up in the nav bar to the left (and subsequently block the
*actually important* things from appearing).
Differential Revision: https://phabricator.services.mozilla.com/D16254
--HG--
extra : moz-landing-system : lando
It turns out that Python's `datetime.isoformat` method will leave off the
fractional component of seconds if it would be all zeroes, but the voluptuous
`Datetime` validator wants it to be present, so it's possible to hit an error
if you run mach at exactly an integer second.
This patch switches from `isoformat` to `strftime` with an explicit format
string instead.
Differential Revision: https://phabricator.services.mozilla.com/D15981
--HG--
extra : moz-landing-system : lando
We have a few bugs filed as intermittent failures all relating
to an `AssertionError` on the return value of this block, which
seems to have a relevant value when `OSError` is thrown. Removing
this will not fix the problem but will at least let us see the
true cause of the failure for debugging and stabilization.
Differential Revision: https://phabricator.services.mozilla.com/D11884
--HG--
extra : moz-landing-system : lando
This change tries to ensure that we don't write telemetry data for mach
commands invoked recursively as part of other mach commands. The intent of
build system telemetry is to only collect data about commands that users are
invoking directly.
There are two ways that we found mach commands can be recursively invoked:
* By running a python subprocess to recursively invoke mach (used in
`mach bootstrap` to call `mach artifact toolchain`)
* By using `Registrar.dispatch` to delegate to a sub-command (used by many
build system commands to invoke `mach build`).
The subprocess case is handled here by having mach set a `MACH_MAIN_PID`
environment variable whose value is the current process' pid on startup if it
does not already exist in the environment. Telemetry code then checks that the
value of that variable matches the current pid and skips writing telemetry data
if not.
The dispatch case is handled by making `MachRegistrar` store the current depth
of the command stack and pass it to the `post_dispatch_handler` which will skip
writing telemetry data if depth != 1.
Additionally the `should_skip_dispatch` function in mach_bootstrap is renamed
to `should_skip_telemetry_submission`, which was its original intent. The
combination of checks added in this change should be sufficient for deciding
when to write telemetry data, and we were not collecting telemetry for the set
of mach commands in that function (which included `mach bootstrap`).
In order to facilitate writing a test for the dispatch case this change adds a
`mach python --exec-file` option to execute Python code directly in the context
of the `mach python` command.
Differential Revision: https://phabricator.services.mozilla.com/D11207
--HG--
extra : moz-landing-system : lando
The build telemetry code attempts to filter paths to avoid PII from usernames
and other things. It does this by converting every commandline argument to
an absolute path and then making them relative to topsrcdir or topobjdir and
omitting any that fail. This meant that running a mach command with a cwd
outside of the topsrcdir or objdir would omit all arguments since they were
converted to absolute paths from the cwd.
This change fixes this by adding the cwd to the list of paths used to create
relative paths. Additionally we add the user's home directory to that list
to try to avoid usernames sneaking through. Finally, instead of simply
removing these path prefixes, we replace them with sigils: $topsrcdir,
$objdir, $HOME.
Differential Revision: https://phabricator.services.mozilla.com/D11174
--HG--
extra : moz-landing-system : lando
This change adds python/mach/mach/test/test_telemetry.py which contains
a simple test that running mach with the build telemetry setting enabled
causes us to write a telemetry file. The test fixture in the file should
make it easy to write additional tests.
A necessary precursor to make the tests work was to change mach_bootstrap's
`should_skip_dispatch` function, which would refuse to write telemetry data
if stdout was not a terminal (which it isn't in the tests) and also in
automation. The latter test is moved to ensure that we don't *submit*
telemetry data from automation, but we can still write it to disk. Machines
in automation should never have the telemetry setting enabled outside of
these tests anyway, so this should not change anything in practice.
Differential Revision: https://phabricator.services.mozilla.com/D11172
--HG--
extra : moz-landing-system : lando
Changes:
- under a specific code path such as:
`./mach <test_type> --debugger`
there exists now additional checks for IndexError when attempting to parse the debugger program name.
- print a nicer error message if user failed to properly specify the debugger program name.
- raise exit code of 1 if IndexError is raised.
Others:
- code comments updated to better reflect/describe what the section does.
Differential Revision: https://phabricator.services.mozilla.com/D10987
--HG--
extra : moz-landing-system : lando
Behavior changes:
- instead of reading the mozconfig file, it now instantiates an instance of the build object.
- safe checking methods are used to access attributes to prevent errors on automation environment.
- better mach command parsing is performed with handler category instead of error-prone argv parsing.
Other changes:
- docstring for testing/xpcshell/runxpcshelltests.py::buildTestList() added and modernized.
- added clause that if length of tests gathered is 0, mach exits with an error code of 1.
Differential Revision: https://phabricator.services.mozilla.com/D7133
--HG--
extra : moz-landing-system : lando
blessings.tigetstr is not part of its API. It happens to work because
blessings imports curses using 'from curses import tigetstr'.
Instead, we can just use terminal.normal, which contains the string we were
going to get anyway.
See https://github.com/erikrose/blessings/pull/138 for more information.
Let me know if there's a better way of resolving this. Hopefully with this +
the patch I submitted to blessings (https://github.com/erikrose/blessings/pull/137)
firefox will build fine with TERM improperly set.
Differential Revision: https://phabricator.services.mozilla.com/D5377
--HG--
extra : moz-landing-system : lando
- added checkers in python/mach/mach/main.py prior to calling registrar.py.
- added internal function to check if specified debugger is installed.
- support both ./mach test <test_name> and ./mach <test_category> styles.
Differential Revision: https://phabricator.services.mozilla.com/D7234
--HG--
extra : moz-landing-system : lando
- added checkers in python/mach/mach/main.py prior to calling registrar.py.
- added internal function to check if specified debugger is installed.
- support both ./mach test <test_name> and ./mach <test_category> styles.
Differential Revision: https://phabricator.services.mozilla.com/D7234
--HG--
extra : moz-landing-system : lando
This will make sure that when running |mach python-test --python 3| locally,
we only run the tests that also run in CI with python 3 (and therefore pass
presumably).
MozReview-Commit-ID: 3OBr9yLSlSq
--HG--
extra : rebase_source : 456340d0ecdddf1078f2b5b4ebb1eddf3813b26a
This patch allows executing |mach python-test| against Python 3 by specifying the optional |--three| command line option. When this option is present, pipenv will be used to manage a virtual environment using Python 3 and attempt to run the tests. When it is not present, pipenv will not be used, and everything will work as it did before this patch.
My original plan was to use pipenv regardless of the target version of Python, however I encountered several issues running some of the tests against our Python packages. Once all tests have been patched to run against Python 3, then we should be able to use pipenv when running them against Python 2.
Note that this patch allows tests to run against Python 3, but there are plenty of issues preventing them from passing. With this patch in place we can start to add Python 3 support to our packages and have the tests running in CI to ensure we don't regress back to just supporting Python 2.
MozReview-Commit-ID: BuU5gZK83hL
IHG: changed taskcluster/ci/source-test/python.yml
--HG--
extra : rebase_source : ca2b15d905f7a5c895a2fd8916144841f5d205de
Edwin Takahashi
Anmol Agarwal
Andrew Halberstadt
championshuttler
Justin Wood
Nick Alexander
Manish Goregaokar
Bobby Holley
Chris Manchester
Noemi Erli
Ted Mielczarek
Connor Sheehan
Jay Kamat
shindli
Dave Hunt