We have a few bugs filed as intermittent failures all relating
to an `AssertionError` on the return value of this block, which
seems to have a relevant value when `OSError` is thrown. Removing
this will not fix the problem but will at least let us see the
true cause of the failure for debugging and stabilization.
Differential Revision: https://phabricator.services.mozilla.com/D11884
--HG--
extra : moz-landing-system : lando
This change tries to ensure that we don't write telemetry data for mach
commands invoked recursively as part of other mach commands. The intent of
build system telemetry is to only collect data about commands that users are
invoking directly.
There are two ways that we found mach commands can be recursively invoked:
* By running a python subprocess to recursively invoke mach (used in
`mach bootstrap` to call `mach artifact toolchain`)
* By using `Registrar.dispatch` to delegate to a sub-command (used by many
build system commands to invoke `mach build`).
The subprocess case is handled here by having mach set a `MACH_MAIN_PID`
environment variable whose value is the current process' pid on startup if it
does not already exist in the environment. Telemetry code then checks that the
value of that variable matches the current pid and skips writing telemetry data
if not.
The dispatch case is handled by making `MachRegistrar` store the current depth
of the command stack and pass it to the `post_dispatch_handler` which will skip
writing telemetry data if depth != 1.
Additionally the `should_skip_dispatch` function in mach_bootstrap is renamed
to `should_skip_telemetry_submission`, which was its original intent. The
combination of checks added in this change should be sufficient for deciding
when to write telemetry data, and we were not collecting telemetry for the set
of mach commands in that function (which included `mach bootstrap`).
In order to facilitate writing a test for the dispatch case this change adds a
`mach python --exec-file` option to execute Python code directly in the context
of the `mach python` command.
Differential Revision: https://phabricator.services.mozilla.com/D11207
--HG--
extra : moz-landing-system : lando
The build telemetry code attempts to filter paths to avoid PII from usernames
and other things. It does this by converting every commandline argument to
an absolute path and then making them relative to topsrcdir or topobjdir and
omitting any that fail. This meant that running a mach command with a cwd
outside of the topsrcdir or objdir would omit all arguments since they were
converted to absolute paths from the cwd.
This change fixes this by adding the cwd to the list of paths used to create
relative paths. Additionally we add the user's home directory to that list
to try to avoid usernames sneaking through. Finally, instead of simply
removing these path prefixes, we replace them with sigils: $topsrcdir,
$objdir, $HOME.
Differential Revision: https://phabricator.services.mozilla.com/D11174
--HG--
extra : moz-landing-system : lando
This change adds python/mach/mach/test/test_telemetry.py which contains
a simple test that running mach with the build telemetry setting enabled
causes us to write a telemetry file. The test fixture in the file should
make it easy to write additional tests.
A necessary precursor to make the tests work was to change mach_bootstrap's
`should_skip_dispatch` function, which would refuse to write telemetry data
if stdout was not a terminal (which it isn't in the tests) and also in
automation. The latter test is moved to ensure that we don't *submit*
telemetry data from automation, but we can still write it to disk. Machines
in automation should never have the telemetry setting enabled outside of
these tests anyway, so this should not change anything in practice.
Differential Revision: https://phabricator.services.mozilla.com/D11172
--HG--
extra : moz-landing-system : lando
Changes:
- under a specific code path such as:
`./mach <test_type> --debugger`
there exists now additional checks for IndexError when attempting to parse the debugger program name.
- print a nicer error message if user failed to properly specify the debugger program name.
- raise exit code of 1 if IndexError is raised.
Others:
- code comments updated to better reflect/describe what the section does.
Differential Revision: https://phabricator.services.mozilla.com/D10987
--HG--
extra : moz-landing-system : lando
Behavior changes:
- instead of reading the mozconfig file, it now instantiates an instance of the build object.
- safe checking methods are used to access attributes to prevent errors on automation environment.
- better mach command parsing is performed with handler category instead of error-prone argv parsing.
Other changes:
- docstring for testing/xpcshell/runxpcshelltests.py::buildTestList() added and modernized.
- added clause that if length of tests gathered is 0, mach exits with an error code of 1.
Differential Revision: https://phabricator.services.mozilla.com/D7133
--HG--
extra : moz-landing-system : lando
blessings.tigetstr is not part of its API. It happens to work because
blessings imports curses using 'from curses import tigetstr'.
Instead, we can just use terminal.normal, which contains the string we were
going to get anyway.
See https://github.com/erikrose/blessings/pull/138 for more information.
Let me know if there's a better way of resolving this. Hopefully with this +
the patch I submitted to blessings (https://github.com/erikrose/blessings/pull/137)
firefox will build fine with TERM improperly set.
Differential Revision: https://phabricator.services.mozilla.com/D5377
--HG--
extra : moz-landing-system : lando
- added checkers in python/mach/mach/main.py prior to calling registrar.py.
- added internal function to check if specified debugger is installed.
- support both ./mach test <test_name> and ./mach <test_category> styles.
Differential Revision: https://phabricator.services.mozilla.com/D7234
--HG--
extra : moz-landing-system : lando
- added checkers in python/mach/mach/main.py prior to calling registrar.py.
- added internal function to check if specified debugger is installed.
- support both ./mach test <test_name> and ./mach <test_category> styles.
Differential Revision: https://phabricator.services.mozilla.com/D7234
--HG--
extra : moz-landing-system : lando
This will make sure that when running |mach python-test --python 3| locally,
we only run the tests that also run in CI with python 3 (and therefore pass
presumably).
MozReview-Commit-ID: 3OBr9yLSlSq
--HG--
extra : rebase_source : 456340d0ecdddf1078f2b5b4ebb1eddf3813b26a
This patch allows executing |mach python-test| against Python 3 by specifying the optional |--three| command line option. When this option is present, pipenv will be used to manage a virtual environment using Python 3 and attempt to run the tests. When it is not present, pipenv will not be used, and everything will work as it did before this patch.
My original plan was to use pipenv regardless of the target version of Python, however I encountered several issues running some of the tests against our Python packages. Once all tests have been patched to run against Python 3, then we should be able to use pipenv when running them against Python 2.
Note that this patch allows tests to run against Python 3, but there are plenty of issues preventing them from passing. With this patch in place we can start to add Python 3 support to our packages and have the tests running in CI to ensure we don't regress back to just supporting Python 2.
MozReview-Commit-ID: BuU5gZK83hL
IHG: changed taskcluster/ci/source-test/python.yml
--HG--
extra : rebase_source : ca2b15d905f7a5c895a2fd8916144841f5d205de
This patch allows executing |mach python-test| against Python 3 by specifying the optional |--three| command line option. When this option is present, pipenv will be used to manage a virtual environment using Python 3 and attempt to run the tests. When it is not present, pipenv will not be used, and everything will work as it did before this patch.
My original plan was to use pipenv regardless of the target version of Python, however I encountered several issues running some of the tests against our Python packages. Once all tests have been patched to run against Python 3, then we should be able to use pipenv when running them against Python 2.
Note that this patch allows tests to run against Python 3, but there are plenty of issues preventing them from passing. With this patch in place we can start to add Python 3 support to our packages and have the tests running in CI to ensure we don't regress back to just supporting Python 2.
MozReview-Commit-ID: BuU5gZK83hL
IHG: changed taskcluster/ci/source-test/python.yml
--HG--
extra : rebase_source : e1a64c0ffa8fe5cce71a041579601d4a72e37779
This is a new issue that gets linted with flake8 3.5.0. Basically you should
never use a blank except: statement.
This will catch all exceptions, including KeyboardInterrupt and SystemExit
(which is likely not intended). If a catch all is needed, use
`except: Exception`. If you *really* mean to also catch KeyboardInterrupt et
al, use `except: BaseException`.
Of course, being specific is often better than a catch all.
MozReview-Commit-ID: FKx80MLO4RN
--HG--
extra : rebase_source : 7c74a7d0d81f2c984b47aff3a0ee3448b791177b
Currently, marking a logger as a structured logger will require a
subsequent function call in order for the logger to be hooked up
to active handlers. This behavior is not intuitive and makes it
easy to not have handlers for newly-registered loggers. This means
messages may not be logged anywhere.
In addition, we have to manually specify which named loggers to
enable structured logging for. This can be annoying.
We change the behavior of register_structured_logger() to
automatically add existing terminal and json handlers to the
logger being marked as structured.
We also introduce an API to enable structured logging for all
loggers. Existing consumers of registered_structured_logger()
in mozbuild have been updated to use this API. A new consumer
has been added for the `mach configure` command because it should
have been there before.
We stop short of making enable_all_structured_loggers() the default.
This is because various commands interact with the log manager in
ways that will result in duplicate logging of messages and
dropping of structured messages. There is a bit of a rabbit hole
here and addressing it can be done as a follow-up.
MozReview-Commit-ID: 1aU6eJvTSMP
--HG--
extra : rebase_source : 2a0a569b378cc3083b55fc7076b291abdfb7453f
We don't use this attribute outside of this function. Besides, you can
easily obtain a handle on the logger by calling
``logging.getLogger('mach')``.
MozReview-Commit-ID: 41vdn6McowW
--HG--
extra : rebase_source : dca383e5f5b770fa3adb09d753897633258302a1
This passes `python3 -mcompileall`. Changes:
* use `0o` prefix for octal literals
* print as a function
* except .. as
* use six.reraise to replace a multi-argument raise statement
* use six.string_types and six.moves.configparser
* remove uses of `L` suffix for long integers
MozReview-Commit-ID: KLaYRNHGpay
--HG--
extra : rebase_source : 6ca1b5447cd28eff8d9f2805add6a0f07e8b4c63
This preserves ./mach settings' --list option. If --list is passed in, we call splitlines()
on the description and print only the first line. The full multi-line description will be
printed otherwise.
This also displays the type and/or choices of the option as appropriate.
MozReview-Commit-ID: 7UMsN9qslWt
--HG--
extra : rebase_source : 4bc9554d8652e02e290c6a190634f1a72cdbadc3
We currently raise if we detect a command has both the `parser` attribute and a subcommand
at the same time, but as far as I can tell, there's no good reason to do this. Handling a
parser + subcommands seems to work exactly how you would expect. Furthermore, it isn't an
error to have subcommands + @CommandArgument, so it doesn't make sense that we're allowing
one but not the other.
This change solves an (admittedly unique) use case I'm trying to build into |mach try|. There
are N subcommands that all support a --save and --load style argument. So, e.g, we might have:
./mach try syntax --save foo
./mach try fuzzy --save bar
The main command will have the ability to detect which subcommand a saved value was generated
from and automatically dispatch to it. So this will work:
./mach try --load foo # dispatches to the syntax subcommand
./mach try --load bar # dispatches to the fuzzy subcommand
In order to share the --save/--load arguments across the main command + subcommands, we need
to set the parser attribute.
MozReview-Commit-ID: KmXRj8TBvYK
--HG--
extra : rebase_source : cbf1e402a080913709a34430274ae3191821dd72
This will facilitate handling this error condition specially.
Some unused imports were also removed.
MozReview-Commit-ID: 4zxVDgE7NxU
--HG--
extra : rebase_source : c4d0dd96629e028ce8a83215564a8d865e4b4b3d
Prefs can be set with `./mach run --setpref foo=bar` or in the ~/.mozbuild/machrc file as:
[runprefs]
foo=bar
MozReview-Commit-ID: HO3tdFi9ffi
--HG--
extra : rebase_source : 3c2e6f33567448c19defafb656e6cb9f3a729391
This allows subcommands to use external argument parsers.
MozReview-Commit-ID: 7TkbTff0Tv8
--HG--
extra : rebase_source : a1c245efa7ac7b28b974534b4cd2727c96f9219d
After this patch, the following will all display the subcommand help where they previously displayed
the command help:
$ mach help <command> <subcommand>
$ mach <command> --help <subcommand>
$ mach <command> <subcommand> --help
The command help will still be shown for:
$ mach help <command>
$ mach <command> --help
MozReview-Commit-ID: 7EsVblnCaFM
--HG--
extra : rebase_source : 2a1d289d56164366ce140fa653adec93f56be067
The sgr0 terminal capabity defines the byte sequence needed to reset
terminal text to its default state. For reasons documented inline in
this commit, we now print this sequence after every line printed by
mach's terminal logger.
MozReview-Commit-ID: 3RukP0QXtqy
--HG--
extra : rebase_source : 5e4b7d001300ec1059b53423b310ac9fdd514c72
This deprecates PYTHON_UNIT_TESTS and replaces it with PYTHON_UNITTEST_MANIFESTS.
In the build system, this means python unittests will be treated the same as all
other test suites that use manifestparser. New manifests called 'python.ini' have
been created for all test directories containing python unittests.
MozReview-Commit-ID: IBHG7Thif2D
--HG--
extra : rebase_source : 11a92a2bc544d067946bbd774975140147458caa
The way it works now, `mach` commands often invoke subprocesses where
the subprocesses' stdio file descriptors are pipes so the mach command
can e.g. parse output.
Processes like clang, gcc, and cargo determine if they can send color
codes to {stderr, stdout} by seeing if those file descriptors are TTYs.
When e.g. `make` is executed via `mach`, this test fails because those
descriptors are pipes (even though they eventually end up on a TTY).
We can't wire the file descriptors to the TTY because `mach` needs
to analyze output. We don't want users defining process flags to force
color in their mozconfigs because color codes would still be sent
if stdout was not a TTY.
This patch sets the MACH_STDOUT_ISATTY environment variable in all mach
commands when stdout is a TTY. Subsequent processes can then look for
this variable to determine whether to override color settings, print
terminal control codes, etc.
MozReview-Commit-ID: GxXP2mQssjC
--HG--
extra : rebase_source : 4b99547b453cb7dd5cb590a71ed554ce2bc4759d
Currently, environment variables set when running mach commands will
propagate after the command is finished. This can allow unwanted state
to bleed through.
This likely isn't an issue today, but isolating state during code
execution is generally a good practice. So do that.
MozReview-Commit-ID: AdaomGub5EF
--HG--
extra : rebase_source : ce81987a1f6de3a16bce6a9e45b9dc8e8eb29b4b
Before, mach would only detect MINGW32 (used with msys). Now it also detects MINGW64 (used with msys2).
MozReview-Commit-ID: 6IrNNmkLKgB
--HG--
extra : rebase_source : e84ee16b5bbc1b6b113163ba5b61759f763645c1
MozReview-Commit-ID: AQ3w2oCPQeN
--HG--
extra : topic : bar
extra : rebase_source : 2e0de1688fca11bd62c88bd66ef9cf9ad97206f8
extra : amend_source : 45c025594d645a9252989c8f13387de05e49d4dd
These config options can be defined in ~/.mozbuild/machrc or topsrcdir/machrc.
Aliases work similar to the identically named option in an hgrc.
For example:
[alias]
browser-test = mochitest -f browser
mochitest = mochitest -f plain
MozReview-Commit-ID: CnOocEslRUI
--HG--
extra : rebase_source : 2a6fa154aca7fea8f159ed840728951a37bc52ec
Runtime configs have been implemented for awhile, but disabled. This patch
enables configuration. Config files will be loaded in the following order
(later files override earlier ones):
1a. $MACHRC
1b. $MOZBUILD_STATE_PATH/machrc (if $MACHRC is unset)
2. topsrcdir/machrc
3. CLI via --settings
Note: .machrc may be used instead of machrc if desired.
MozReview-Commit-ID: IntONAZLGML
--HG--
extra : rebase_source : ff79b129eaea7cca5064d30fa6ddc76fceb9669b
This adds a |mach settings locale-gen| subcommand to automatically generate locale
specific documentation for settings. It also refactors |mach settings-create| to
|mach settings| and moves |mach settings| to |mach settings -l|. Finally it performs
some misc cleanup mostly related to locales.
MozReview-Commit-ID: 1VWLcb9ehAH
--HG--
extra : rebase_source : 8f580217123d79e66323ca4be948a3297ae4ced3
Some sections should support user-defined options. For example, in an [alias] section, the option names
are not well-defined, rather specified by the user. This patch allows user-defined option names for any
section that has a 'section.*' option defined. Even with 'section.*', option types are still well-defined.
MozReview-Commit-ID: L34W9v9Fy28
--HG--
extra : rebase_source : 9333f552edead9bf1cf464e28ef8fbbb9bed5597
Defining settings was a little complicated. First it required overriding a '_register_settings'
method, and then it required making N calls to a second 'register_setting' method within that.
This patch simplifies the process of defining settings by only requiring a
'config_settings' attribute. This attribute should be a list of tuples:
[
('<section>.<option>', '<type>', <default>, set(<choices)),
]
`default` and `choices` are optional. Alternatively, 'config_settings' can be a callable
that returns a list of tuples in the same format. This still allows for greater flexibility
for more advanced cases.
MozReview-Commit-ID: F4DTTNJdJsa
--HG--
extra : rebase_source : e3dd455ba559cd3992c9c1b3eaf021c9e0707cc1
For example, say there is a command 'foo' that has a subcommand 'bar'. Prior to this, it was not
possible to run:
./mach foo
as its own independent command. The above would instead print the subcommand help for 'bar'.
MozReview-Commit-ID: JU4dXoxnCyu
--HG--
extra : rebase_source : bb15532ad39456b270071bc60d7b15e15af04e48
This is currently preventing a command from having both args and subcommands at the same
time.
MozReview-Commit-ID: 66frAqamGjv
--HG--
extra : rebase_source : 3bbd1cb508e5aab7ba7f9936c0c486ed8f626fe9
Connor Sheehan
Ted Mielczarek
Edwin Gao
Chris Manchester
Jay Kamat
shindli
Andrew Halberstadt
Dave Hunt
Andreea Pavel
David Major
Tarek Ziadé
Gregory Szorc
Steve Armand
Dustin J. Mitchell
Jonas Finnemann Jensen
Brian Grinstead
Michael Layzell
Nathan Hakkakzadeh
Steve Fink