Bug 1543247 - Part 1: Add `mach browsertime` command that installs and invokes browsertime. r=Standard8,ahal

[browsertime](https://github.com/sitespeedio/browsertime) is a harness
for running performance tests, similar to Mozilla's Raptor testing
framework.  The Performance Team is using it locally with some
success, but we're running a heavily modified toolchain that is
challenging to install.  This mach command is intended to be leverage
for getting more folks able to use browsertime easily.

In particular, the version of browsertime that this installs has
nalexander's changes to support testing GeckoView-based vehicles.  If
this approach meets with approval, I'll continue to follow-up with
additional configuration and tooling layers to make it even easier to
drive GeckoView-based vehicles.

I elected to piggy-back install on the eslint installation process,
since this is very similar.  To that end, I generalized what was there
very slightly.  I elected not to try to move the existing code into a
more obvious shared location, although it might be possible, because
it wasn't clear what contexts the existing code would be invoked
from.  In particular I wasn't certain the code could rely on a
complete mozbuild checkout.

I did need to ensure the local Node.js binary is early on the PATH;
this was an issue I ran into with my initial Node/Yarn prototyping
many months ago.  At heart the issue is that package scripts in the
wild invoke a bare `node` or `npm` command; if there was a culture of
invoking $NODE or $NPM, this wouldn't be necessary.  There's no harm
doing it for ESlint, and it will help the next person who wants to
install an NPM package for tooling in this manner.

Differential Revision: https://phabricator.services.mozilla.com/D26820

--HG--
extra : moz-landing-system : lando

.hgignore

@@ -129,8 +129,9 @@ _OPT\.OBJ/
 # Ignore tox generated dir
 .tox/
 
-# Ignore ESLint node_modules
+# Ignore ESLint and other tool's node_modules.
 ^node_modules/
+^tools/browsertime/node_modules/
 ^tools/lint/eslint/eslint-plugin-mozilla/node_modules/
 
 # Ignore talos virtualenv and tp5n files.

build/mach_bootstrap.py

@@ -61,6 +61,7 @@ MACH_MODULES = [
     'testing/web-platform/mach_commands.py',
     'testing/xpcshell/mach_commands.py',
     'toolkit/components/telemetry/tests/marionette/mach_commands.py',
+    'tools/browsertime/mach_commands.py',
     'tools/compare-locales/mach_commands.py',
     'tools/docs/mach_commands.py',
     'tools/lint/mach_commands.py',

tools/browsertime/mach_commands.py

@@ -0,0 +1,105 @@
+# This Source Code Form is subject to the terms of the Mozilla Public
+# License, v. 2.0. If a copy of the MPL was not distributed with this
+# file, You can obtain one at http://mozilla.org/MPL/2.0/.
+
+r'''Make it easy to install and run [browsertime](https://github.com/sitespeedio/browsertime).
+
+Browsertime is a harness for running performance tests, similar to
+Mozilla's Raptor testing framework.  Browsertime is written in Node.js
+and uses Selenium WebDriver to drive multiple browsers including
+Chrome, Chrome for Android, Firefox, and (pending the resolution of
+[Bug 1525126](https://bugzilla.mozilla.org/show_bug.cgi?id=1525126)
+and similar tickets) Firefox for Android and GeckoView-based vehicles.
+
+Right now a custom version of browsertime and the underlying
+geckodriver binary are needed to support GeckoView-based vehicles;
+this module accommodates those in-progress custom versions.
+
+To get started, run
+```
+./mach browsertime --setup [--clobber]
+```
+This will populate `tools/browsertime/node_modules`.
+
+To invoke browsertime, run
+```
+./mach browsertime [ARGS]
+```
+All arguments are passed through to browsertime.
+'''
+
+from __future__ import absolute_import, print_function, unicode_literals
+
+import argparse
+import logging
+import os
+import sys
+
+import mozpack.path as mozpath
+from mach.decorators import CommandArgument, CommandProvider, Command
+from mozbuild.base import MachCommandBase
+from mozbuild.nodeutil import find_node_executable
+
+sys.path.append(mozpath.join(os.path.dirname(__file__), '..', '..', 'tools', 'lint', 'eslint'))
+import setup_helper
+
+
+@CommandProvider
+class MachBrowsertime(MachCommandBase):
+    def setup(self, should_clobber=False):
+        if not setup_helper.check_node_executables_valid():
+            return 1
+
+        return setup_helper.package_setup(
+            os.path.dirname(__file__),
+            'browsertime',
+            should_clobber=should_clobber)
+
+    def node(self, args):
+        node, _ = find_node_executable()
+
+        # Ensure that bare `node` and `npm` in scripts, including post-install
+        # scripts, finds the binary we're invoking with.  Without this, it's
+        # easy for compiled extensions to get mismatched versions of the Node.js
+        # extension API.
+        path = os.environ.get('PATH', '').split(os.pathsep)
+        node_path = os.path.dirname(node)
+        if node_path not in path:
+            path = [node_path] + path
+
+        return self.run_process(
+            [node] + args,
+            append_env={'PATH': os.pathsep.join(path)},
+            pass_thru=True,  # Allow user to run Node interactively.
+            ensure_exit_code=False,  # Don't throw on non-zero exit code.
+            cwd=mozpath.join(self.topsrcdir))
+
+    def bin_path(self):
+        # On Windows, invoking `node_modules/.bin/browsertime{.cmd}`
+        # doesn't work when invoked as an argument to our specific
+        # binary.  Since we want our version of node, invoke the
+        # actual script directly.
+        return mozpath.join(
+            os.path.dirname(__file__),
+            'node_modules',
+            'browsertime',
+            'bin',
+            'browsertime.js')
+
+    @Command('browsertime', category='testing',
+             description='Run [browsertime](https://github.com/sitespeedio/browsertime) '
+                         'performance tests.')
+    @CommandArgument('--verbose', action='store_true',
+                     help='Verbose output for what commands the build is running.')
+    @CommandArgument('--setup', default=False, action='store_true')
+    @CommandArgument('--clobber', default=False, action='store_true')
+    @CommandArgument('args', nargs=argparse.REMAINDER)
+    def browsertime(self, args, verbose=False, setup=False, clobber=False):
+        if setup:
+            return self.setup(should_clobber=clobber)
+
+        if not verbose:
+            # Avoid logging the command
+            self.log_manager.terminal_handler.setLevel(logging.CRITICAL)
+
+        return self.node([self.bin_path()] + args)

tools/browsertime/package.json

@@ -0,0 +1,12 @@
+{
+  "name": "mozilla-central-tools-browsertime",
+  "description": "This package file is for node modules used in mozilla-central/tools/browsertime",
+  "repository": {},
+  "license": "MPL-2.0",
+  "dependencies": {},
+  "devDependencies": {
+    "browsertime": "github:ncalexan/browsertime#06c82fa99daafee70f10b0401fc6bc38f77cdee4"
+  },
+  "notes(private)": "We don't want to publish to npm, so this is marked as private",
+  "private": true
+}

tools/lint/eslint/setup_helper.py

@@ -71,73 +71,98 @@ def eslint_setup(should_clobber=False):
     guide you through an interactive wizard helping you configure
     eslint for optimal use on Mozilla projects.
     """
+    package_setup(get_project_root(), 'eslint', should_clobber=should_clobber)
+
+
+def package_setup(package_root, package_name, should_clobber=False):
+    """Ensure `package_name` at `package_root` is installed.
+
+    This populates `package_root/node_modules`.
+    """
+    orig_project_root = get_project_root()
     orig_cwd = os.getcwd()
-    sys.path.append(os.path.dirname(__file__))
+    try:
+        set_project_root(package_root)
+        sys.path.append(os.path.dirname(__file__))
 
-    # npm sometimes fails to respect cwd when it is run using check_call so
-    # we manually switch folders here instead.
-    project_root = get_project_root()
-    os.chdir(project_root)
+        # npm sometimes fails to respect cwd when it is run using check_call so
+        # we manually switch folders here instead.
+        project_root = get_project_root()
+        os.chdir(project_root)
 
-    if should_clobber:
-        node_modules_path = os.path.join(project_root, "node_modules")
-        print("Clobbering node_modules...")
-        if sys.platform.startswith('win') and have_winrm():
-            process = subprocess.Popen(['winrm', '-rf', node_modules_path])
-            process.wait()
+        if should_clobber:
+            node_modules_path = os.path.join(project_root, "node_modules")
+            print("Clobbering %s..." % node_modules_path)
+            if sys.platform.startswith('win') and have_winrm():
+                process = subprocess.Popen(['winrm', '-rf', node_modules_path])
+                process.wait()
+            else:
+                mozfileremove(node_modules_path)
+
+        npm_path, version = find_npm_executable()
+        if not npm_path:
+            return 1
+
+        node_path, _ = find_node_executable()
+        if not node_path:
+            return 1
+
+        extra_parameters = ["--loglevel=error"]
+
+        package_lock_json_path = os.path.join(get_project_root(), "package-lock.json")
+        package_lock_json_tmp_path = os.path.join(tempfile.gettempdir(), "package-lock.json.tmp")
+
+        # If we have an npm version newer than 5.8.0, just use 'ci', as that's much
+        # simpler and does exactly what we want.
+        npm_is_older_version = version < StrictVersion("5.8.0").version
+
+        if npm_is_older_version:
+            cmd = [npm_path, "install"]
+            shutil.copy2(package_lock_json_path, package_lock_json_tmp_path)
         else:
-            mozfileremove(node_modules_path)
+            cmd = [npm_path, "ci"]
 
-    npm_path, version = find_npm_executable()
-    if not npm_path:
-        return 1
+        # On non-Windows, ensure npm is called via node, as node may not be in the
+        # path.
+        if platform.system() != "Windows":
+            cmd.insert(0, node_path)
 
-    node_path, _ = find_node_executable()
-    if not node_path:
-        return 1
+        cmd.extend(extra_parameters)
 
-    extra_parameters = ["--loglevel=error"]
+        # Ensure that bare `node` and `npm` in scripts, including post-install scripts, finds the
+        # binary we're invoking with.  Without this, it's easy for compiled extensions to get
+        # mismatched versions of the Node.js extension API.
+        path = os.environ.get('PATH', '').split(os.pathsep)
+        node_dir = os.path.dirname(node_path)
+        if node_dir not in path:
+            path = [node_dir] + path
 
-    package_lock_json_path = os.path.join(get_project_root(), "package-lock.json")
-    package_lock_json_tmp_path = os.path.join(tempfile.gettempdir(), "package-lock.json.tmp")
+        print("Installing %s for mach using \"%s\"..." % (package_name, " ".join(cmd)))
+        result = call_process(package_name, cmd, append_env={'PATH': os.pathsep.join(path)})
 
-    # If we have an npm version newer than 5.8.0, just use 'ci', as that's much
-    # simpler and does exactly what we want.
-    npm_is_older_version = version < StrictVersion("5.8.0").version
+        if npm_is_older_version:
+            shutil.move(package_lock_json_tmp_path, package_lock_json_path)
 
-    if npm_is_older_version:
-        cmd = [npm_path, "install"]
-        shutil.copy2(package_lock_json_path, package_lock_json_tmp_path)
-    else:
-        cmd = [npm_path, "ci"]
+        if not result:
+            return 1
 
-    # On non-Windows, ensure npm is called via node, as node may not be in the
-    # path.
-    if platform.system() != "Windows":
-        cmd.insert(0, node_path)
+        bin_path = os.path.join(get_project_root(), "node_modules", ".bin", package_name)
 
-    cmd.extend(extra_parameters)
-    print("Installing eslint for mach using \"%s\"..." % (" ".join(cmd)))
-    result = call_process("eslint", cmd)
+        print("\n%s installed successfully!" % package_name)
+        print("\nNOTE: Your local %s binary is at %s\n" % (package_name, bin_path))
 
-    if npm_is_older_version:
-        shutil.move(package_lock_json_tmp_path, package_lock_json_path)
-
-    if not result:
-        return 1
-
-    eslint_path = os.path.join(get_project_root(), "node_modules", ".bin", "eslint")
-
-    print("\nESLint and approved plugins installed successfully!")
-    print("\nNOTE: Your local eslint binary is at %s\n" % eslint_path)
-
-    os.chdir(orig_cwd)
+    finally:
+        set_project_root(orig_project_root)
+        os.chdir(orig_cwd)
 
 
-def call_process(name, cmd, cwd=None):
+def call_process(name, cmd, cwd=None, append_env={}):
+    env = dict(os.environ)
+    env.update(append_env)
+
     try:
         with open(os.devnull, "w") as fnull:
-            subprocess.check_call(cmd, cwd=cwd, stdout=fnull)
+            subprocess.check_call(cmd, cwd=cwd, stdout=fnull, env=env)
     except subprocess.CalledProcessError:
         if cwd:
             print("\nError installing %s in the %s folder, aborting." % (name, cwd))