This logic is very 'mozill-central' specific and should live outside of mach
core if possible. Luckily we already have a concept of a 'pre_dispatch_handler'
that is meant for exactly this type of use case.
Differential Revision: https://phabricator.services.mozilla.com/D47668
--HG--
extra : moz-landing-system : lando
Credit: mars for making the shell POSIX compliant
This embeds a blacklist of every mach command that needs to run with Python 2
directly in the mach driver itself. Initially this is every mach command. We
then use a bit of shell to determine whether the command being run needs Python
2 or 3.
While this approach may seem a bit hacky, it has several benefits:
1. No need to add complex machinery in mach's registration code.
2. No need to spawn two separate Python interpreters in the event a different
Python from the original interpreter is needed.
3. Perf impact is negligible.
4. New commands are Python 3 by default.
It is also only a temporary hack. Once all commands are running with Python 3,
we can revert back to the original mach driver.
Differential Revision: https://phabricator.services.mozilla.com/D36103
--HG--
extra : moz-landing-system : lando
Introduces "./mach remote vendor-puppeteer" for vendoring the
Puppeteer client without dependencies into remote/test/puppeteer/.
The particular checkout of Puppeteer is
https://github.com/andreastt/puppeteer/tree/firefox, which contains a
couple of hotfixes we need for the client to work with the Firefox
implementation of CDP.
The remote agent targets a specific version of Puppeteer, so it is
not suitable for this to be vendored under third_party/. We also
wouldn't want other code in central to accidentally use a patched fork.
The vendoring process is not part of "./mach vendor" because it does
not yet have Node.js support, and implementing that for mach is outside
the scope of getting the Puppeteer tests running with the remote agent.
Differential Revision: https://phabricator.services.mozilla.com/D37007
--HG--
extra : moz-landing-system : lando
Introduces "./mach remote vendor-puppeteer" for vendoring the
Puppeteer client without dependencies into remote/test/puppeteer/.
The particular checkout of Puppeteer is
https://github.com/andreastt/puppeteer/tree/firefox, which contains a
couple of hotfixes we need for the client to work with the Firefox
implementation of CDP.
The remote agent targets a specific version of Puppeteer, so it is
not suitable for this to be vendored under third_party/. We also
wouldn't want other code in central to accidentally use a patched fork.
The vendoring process is not part of "./mach vendor" because it does
not yet have Node.js support, and implementing that for mach is outside
the scope of getting the Puppeteer tests running with the remote agent.
Differential Revision: https://phabricator.services.mozilla.com/D37007
--HG--
extra : moz-landing-system : lando
On Windows in Python 2, the subprocess module requires the use of bytes with
the 'env' argument. For that reason, we would sometimes use byte strings with
'os.environ' like so:
os.environ[b"FOO"] = b"bar"
However, this is a failure with Python 3 as 'os.environ' must only be used with
the text type. This patch creates a new 'setenv' helper that ensures we create
new environment with 'bytes' on Python 2, and 'text' on Python 3.
Differential Revision: https://phabricator.services.mozilla.com/D38237
--HG--
extra : moz-landing-system : lando
This is a simple mach command that imports a PR from a whitelisted set
of github repositories into the local m-c clone. It works by downloading
the .patch file from github, splitting the different commits, and
applying those commits to the local repo via the `patch` tool and git/hg
commit. It optionally allows filing a bug or providing a bug number, and
specifying a reviewer.
This is one part of a larger workflow that facilitates landing
contributor patches into m-c when those patches are submitted as PRs.
Other components of the workflow (to be added in the future) will make
it easier to actually test and land the patch.
Differential Revision: https://phabricator.services.mozilla.com/D35206
--HG--
extra : moz-landing-system : lando
Negative value for import level is obsolete in Python 3, which was used on Py2 for implicit relative import. This change ensures the level value to be >=0 on Py3 to fix test failures.
Differential Revision: https://phabricator.services.mozilla.com/D32497
--HG--
extra : moz-landing-system : lando
[browsertime](https://github.com/sitespeedio/browsertime) is a harness
for running performance tests, similar to Mozilla's Raptor testing
framework. The Performance Team is using it locally with some
success, but we're running a heavily modified toolchain that is
challenging to install. This mach command is intended to be leverage
for getting more folks able to use browsertime easily.
In particular, the version of browsertime that this installs has
nalexander's changes to support testing GeckoView-based vehicles. If
this approach meets with approval, I'll continue to follow-up with
additional configuration and tooling layers to make it even easier to
drive GeckoView-based vehicles.
I elected to piggy-back install on the eslint installation process,
since this is very similar. To that end, I generalized what was there
very slightly. I elected not to try to move the existing code into a
more obvious shared location, although it might be possible, because
it wasn't clear what contexts the existing code would be invoked
from. In particular I wasn't certain the code could rely on a
complete mozbuild checkout.
I did need to ensure the local Node.js binary is early on the PATH;
this was an issue I ran into with my initial Node/Yarn prototyping
many months ago. At heart the issue is that package scripts in the
wild invoke a bare `node` or `npm` command; if there was a culture of
invoking $NODE or $NPM, this wouldn't be necessary. There's no harm
doing it for ESlint, and it will help the next person who wants to
install an NPM package for tooling in this manner.
Differential Revision: https://phabricator.services.mozilla.com/D26820
--HG--
extra : moz-landing-system : lando
[browsertime](https://github.com/sitespeedio/browsertime) is a harness
for running performance tests, similar to Mozilla's Raptor testing
framework. The Performance Team is using it locally with some
success, but we're running a heavily modified toolchain that is
challenging to install. This mach command is intended to be leverage
for getting more folks able to use browsertime easily.
In particular, the version of browsertime that this installs has
nalexander's changes to support testing GeckoView-based vehicles. If
this approach meets with approval, I'll continue to follow-up with
additional configuration and tooling layers to make it even easier to
drive GeckoView-based vehicles.
I elected to piggy-back install on the eslint installation process,
since this is very similar. To that end, I generalized what was there
very slightly. I elected not to try to move the existing code into a
more obvious shared location, although it might be possible, because
it wasn't clear what contexts the existing code would be invoked
from. In particular I wasn't certain the code could rely on a
complete mozbuild checkout.
I did need to ensure the local Node.js binary is early on the PATH;
this was an issue I ran into with my initial Node/Yarn prototyping
many months ago. At heart the issue is that package scripts in the
wild invoke a bare `node` or `npm` command; if there was a culture of
invoking $NODE or $NPM, this wouldn't be necessary. There's no harm
doing it for ESlint, and it will help the next person who wants to
install an NPM package for tooling in this manner.
Differential Revision: https://phabricator.services.mozilla.com/D26820
--HG--
extra : moz-landing-system : lando
Sometimes we want to store state that only applies to a particular srcdir, but
there isn't a standard directory where this lives. Let's add an argument to
'get_state_dir()' to provide an "official" place.
The new API to get the local state dir is 'get_state_dir(srcdir=True)'. Like
the global state dir, this directory is not guaranteed to exist. A reference to
this value can also be obtained via 'self._mach_context.local_state_dir' from
within a mach command (in this case it will be created automatically if it
doesn't exist).
Note: we should probably just make sure both exist at mach startup, but it felt
outside the scope of this change.
Differential Revision: https://phabricator.services.mozilla.com/D15724
--HG--
extra : moz-landing-system : lando
mozboot.util.get_state_dir() returns a tuple of (<path>, <bool). The bool
denotes whether or not the state dir came from an environment variable.
But this value is only used in a single place, and is very easy to test for
anyway. It's not worth the added complexity it imposes on all other consumers
of this function. Let's just make this function return the path.
Differential Revision: https://phabricator.services.mozilla.com/D15723
--HG--
extra : moz-landing-system : lando
Currently, build telemetry submits at random, approximately
every 10 `mach` invocations. This choice was made arbitrarily,
with no real reason in mind for that level of frequency.
After speaking with some of the data engineers in #telemetry,
it seems we should be able to send pings to the telemetry
pipeline far more frequently than we realized. This commit
removes the telemetry submission logic and causes clients
to attempt to send pings for every mach invocation. Pings
are still saved to the outgoing directory, in case of a
failure or in the case of offline `mach` runs.
Differential Revision: https://phabricator.services.mozilla.com/D11279
--HG--
extra : moz-landing-system : lando
While attempting to improve the build telemetry submission
logic, I found a bug in the way telemetry submission
works. Essentially the submission script was failing to
import any of the required packages (specifically
`mozbuild.telemetry` in this case) as the method used to
modify path was incorrect and the script was running outside
of the virtualenv. The invocation is also sending stdout
and stderr to `/dev/null`, making this problem even less obvious.
When I fixed the path modifications, I realized that `mozbuild`
imports will require a long chain of other imports
(and transitively, more `sys.path` modifications)
such as `which`, `mach`, `mozautomation`, etc to complete.
When I tested the submission script, I did so by running
`mach python build/submit_telemetry_data.py`, which runs the
script in a virtualenv with all required packages installed.
That's likely part of the reasons I overlooked this issue in testing.
Rather than go through the process of importing every dependency
of `mozbuild`, this commit changes the invocation of the submission
script to go through `mach python`. Things seem to work as
expected with this change.
Differential Revision: https://phabricator.services.mozilla.com/D11278
--HG--
extra : moz-landing-system : lando
This change tries to ensure that we don't write telemetry data for mach
commands invoked recursively as part of other mach commands. The intent of
build system telemetry is to only collect data about commands that users are
invoking directly.
There are two ways that we found mach commands can be recursively invoked:
* By running a python subprocess to recursively invoke mach (used in
`mach bootstrap` to call `mach artifact toolchain`)
* By using `Registrar.dispatch` to delegate to a sub-command (used by many
build system commands to invoke `mach build`).
The subprocess case is handled here by having mach set a `MACH_MAIN_PID`
environment variable whose value is the current process' pid on startup if it
does not already exist in the environment. Telemetry code then checks that the
value of that variable matches the current pid and skips writing telemetry data
if not.
The dispatch case is handled by making `MachRegistrar` store the current depth
of the command stack and pass it to the `post_dispatch_handler` which will skip
writing telemetry data if depth != 1.
Additionally the `should_skip_dispatch` function in mach_bootstrap is renamed
to `should_skip_telemetry_submission`, which was its original intent. The
combination of checks added in this change should be sufficient for deciding
when to write telemetry data, and we were not collecting telemetry for the set
of mach commands in that function (which included `mach bootstrap`).
In order to facilitate writing a test for the dispatch case this change adds a
`mach python --exec-file` option to execute Python code directly in the context
of the `mach python` command.
Differential Revision: https://phabricator.services.mozilla.com/D11207
--HG--
extra : moz-landing-system : lando
The build telemetry code attempts to filter paths to avoid PII from usernames
and other things. It does this by converting every commandline argument to
an absolute path and then making them relative to topsrcdir or topobjdir and
omitting any that fail. This meant that running a mach command with a cwd
outside of the topsrcdir or objdir would omit all arguments since they were
converted to absolute paths from the cwd.
This change fixes this by adding the cwd to the list of paths used to create
relative paths. Additionally we add the user's home directory to that list
to try to avoid usernames sneaking through. Finally, instead of simply
removing these path prefixes, we replace them with sigils: $topsrcdir,
$objdir, $HOME.
Differential Revision: https://phabricator.services.mozilla.com/D11174
--HG--
extra : moz-landing-system : lando
After bug 1497638 the method by which build telemetry data gets written
to disk is slightly convoluted. Since we're now invoking `gather_telemetry`
from `post_dispatch_handler`, we just make that function return the data
it gathers and inline the contents of `telemetry_handler` after the call
to it.
Differential Revision: https://phabricator.services.mozilla.com/D11173
--HG--
extra : moz-landing-system : lando
This change adds python/mach/mach/test/test_telemetry.py which contains
a simple test that running mach with the build telemetry setting enabled
causes us to write a telemetry file. The test fixture in the file should
make it easy to write additional tests.
A necessary precursor to make the tests work was to change mach_bootstrap's
`should_skip_dispatch` function, which would refuse to write telemetry data
if stdout was not a terminal (which it isn't in the tests) and also in
automation. The latter test is moved to ensure that we don't *submit*
telemetry data from automation, but we can still write it to disk. Machines
in automation should never have the telemetry setting enabled outside of
these tests anyway, so this should not change anything in practice.
Differential Revision: https://phabricator.services.mozilla.com/D11172
--HG--
extra : moz-landing-system : lando
This patch adds two new actions to virtualenv_packages.txt processing:
windows and !windows. The former processes the rest of the action only on
Windows, and the latter processes it only on non-Windows.
These new features are used in virtualenv_packages.txt to use the
path to the unpacked Windows psutil wheel when on Windows, and build psutil
from source and use that path on other platforms.
This fixes the long-standing problem of not having psutil available on most
Windows systems (since they don't have the right set of Visual C++ build tools).
Differential Revision: https://phabricator.services.mozilla.com/D3437
--HG--
extra : moz-landing-system : lando
The telemetry gathering code is currently split in two places, so move it
all to a common place. Followup patches will rewrite most of this code.
--HG--
extra : rebase_source : afd14940f7175f7ca730a3b90ad770f143bcb6c7
Added `./mach python-safety`, distinct from python-test so it doesn't have
to be run on every CI job - its errors may not depend on the area the push has changed.
Added the python/safety directory to ensure a different Pipfile is used, avoiding
conflicts with python-test.
Differential Revision: https://phabricator.services.mozilla.com/D1825
--HG--
extra : moz-landing-system : lando
This is a new issue that gets linted with flake8 3.5.0. Basically you should
never use a blank except: statement.
This will catch all exceptions, including KeyboardInterrupt and SystemExit
(which is likely not intended). If a catch all is needed, use
`except: Exception`. If you *really* mean to also catch KeyboardInterrupt et
al, use `except: BaseException`.
Of course, being specific is often better than a catch all.
MozReview-Commit-ID: FKx80MLO4RN
--HG--
extra : rebase_source : 7c74a7d0d81f2c984b47aff3a0ee3448b791177b
comm-central taskcluster builds have comm-central checkout in `comm/`. This
will add packages specified in comm-central when it checked out, and will
ignore the missing file for mozilla-central builds.
MozReview-Commit-ID: Amg5vJHoWkS
--HG--
extra : rebase_source : 440e1bab50c2185d18f18f7acaea83a24f0d0aaf
Andrew Halberstadt
Andreas Tolfsen
Ciure Andrei
David Walsh
Andi-Bogdan Postelnicu
Sylvestre Ledru
Cosmin Sabou
Kartikaya Gupta
Kagami Sascha Rosylight
Brindusan Cristian
Nick Alexander
Razvan Maries
Chris Manchester
Noemi Erli
Chris Hartjes
Tom Prince
Ted Mielczarek
Connor Sheehan
Panos Astithas
Simon Fraser
Dave Hunt
Rob Wood
Edouard Oger