As an intermediate step to allow mach commands as standalone functions, the MachCommandBase
subclass instance that currently corresponds to self has to be made available as a separate
argument (named command_context).
Differential Revision: https://phabricator.services.mozilla.com/D109650
Migrates two strings to fluent and uses sentence casing.
Places identity security block into a toolbar button.
Fixes margin spacing.
Removes green color from secure connection.
Differential Revision: https://phabricator.services.mozilla.com/D111368
Migrates two strings to fluent and uses sentence casing.
Places identity security block into a toolbar button.
Fixes margin spacing.
Removes green color from secure connection.
Differential Revision: https://phabricator.services.mozilla.com/D111368
I manually compared `$TOPOBJDIR/faster/unified_install_dist_bin` for
an artifact and a non-artifact build, and found only a few
pre-processing differences. I can't recall why this wasn't supported
initially; I think it used to be that manifests overlapped and that
the interaction with the RecursiveMake backend was more complicated.
But I see no reason why this shouldn't work, and would like to get
some folks to test it in the wild.
Differential Revision: https://phabricator.services.mozilla.com/D110311
***
Bug 1698838 - Restrict the list of linters to valid formatters r=ahal
***
Bug 1698838 - Abort formatting even if one of the linters passed is invalid r=ahal
***
***
Differential Revision: https://phabricator.services.mozilla.com/D109231
This also makes `MOZ_UPDATE_AGENT` depend on `MOZ_BACKGROUNDTASKS`.
There's no need to lint the `tools/update-programs` project, since
it's not a supported configuration. The next consumer can
rehabilitate it or remove it.
Differential Revision: https://phabricator.services.mozilla.com/D111313
Due to strange module-name generation we do (e.g.:
`mach.commands.be68b9824a51408e815262b2e8f21d6b`), Sentry's conversion
of absolute paths to local paths fails - instead, the path remains
absolute, but without the leading forward slash. (Except on Windows,
where `c:/` is maintained).
We resolve this by not requiring the leading slash when patching
absolute paths.
Differential Revision: https://phabricator.services.mozilla.com/D110436
This can occur if the user is using cinnabar and it fails to find the
equivalent hg revision for their current base commit.
It's significantly less valuable to have exceptions with an "anonymous
release", so we skip these exceptions instead.
Differential Revision: https://phabricator.services.mozilla.com/D110481
To track whether issues being reported are for old versions, we attach
an hg revision hash as a release version with each report.
We use the base_ref because:
* It's a public revision that we can look up, and
* Since we only submit Sentry events when mach is unmodified,
it's essentially acting as a version identifier.
Differential Revision: https://phabricator.services.mozilla.com/D108307
AFAIK, while mochitests have resource-usage.json data with the same
format as build_resources.json, there isn't much of anything to display
that data in a human-friendly manner. As a helper of sort for all the
tasks that already expose their json data without a nice way to display
it, allow the build resources usage report UI that `mach resource-usage`
displays (or build_resources.html on build tasks on treeherder), to load
arbitrary json files if their url is dropped on the page (from e.g.
drag-and-dropping the link to the file from treeherder).
Differential Revision: https://phabricator.services.mozilla.com/D109030
This commit adds a simple pageload test, a warm pageload of https://www.google.com.
Low-level perfstats are collected.
All platforms were included. Not running as a cron job.
The motivation is that we can now collect low-level perfStats timings and determine how useful they will be for performance work.
Differential Revision: https://phabricator.services.mozilla.com/D109072
This commit adds the core algorithm for editing moz.build files, tests for a few
pieces of it, and the code to manually run the algorithm over the gecko codebase
and see dynamically how well it performs at trying to assigning existing source
files back into their original locations.
Differential Revision: https://phabricator.services.mozilla.com/D104754
During `mach bootstrap` for Firefox desktop, m4 is installed via `libgtk2.0-dev`.
Since we don't install that package for Android bootstrap, this causes
configure failures.
Install `m4` as a common package to fix configure on Android. Since we plan
to remove `libgtk2.0-dev` soon, this will also avoid breakage when that change
lands.
Differential Revision: https://phabricator.services.mozilla.com/D108692
The build always uses them anyways, but configure tests don't, and some
of them fail as a consequence in some cases (example: when UBsan is
enabled, all AC_LANG_CPLUSPLUS + AC_CHECK_FUNCS tests fail). We also had
similar problems in the past, such as bug 1513605.
By adding the PIC flags to CFLAGS and CXXFLAGS, we ensure old-configure
tests use the flags as well.
While here, we also remove DSO_CFLAGS, which was always empty, and we
stop passing -fPIC to wasm compiles, because it has no effect there.
Differential Revision: https://phabricator.services.mozilla.com/D108560
The build always uses them anyways, but configure tests don't, and some
of them fail as a consequence in some cases (example: when UBsan is
enabled, all AC_LANG_CPLUSPLUS + AC_CHECK_FUNCS tests fail). We also had
similar problems in the past, such as bug 1513605.
By adding the PIC flags to CFLAGS and CXXFLAGS, we ensure old-configure
tests use the flags as well.
While here, we also remove DSO_CFLAGS, which was always empty, and we
stop passing -fPIC to wasm compiles, because it has no effect there.
Differential Revision: https://phabricator.services.mozilla.com/D108560
When Sentry slurps context, it doesn't normalize os path separators for
us. So, we adapt our path aliasing to always catch both forward slashes
and backslashes.
Removed the `needle is None` guard because it because obsolete after
1695366, which removed the case where `topsrcdir` was `None`.
Differential Revision: https://phabricator.services.mozilla.com/D108667
If the "mach" file and all "*.py" files are unchanged, then the reported
exception is likely legitimate and can be reproduced on the public tree.
Differential Revision: https://phabricator.services.mozilla.com/D108262
This patch:
-Maintains consistency between the fxa and remote tabs' panels "sync now" buttons
-Removes a fluent string no longer in use
-Restores tooltip functionality
Differential Revision: https://phabricator.services.mozilla.com/D106651
Bootstrap runs mach create-mach-environment, which may or may not
successfully install glean. When it doesn't, it's not useful to ask
whether to enable telemetry when it's not going to work.
Differential Revision: https://phabricator.services.mozilla.com/D108131
Updates `./mach bootstrap` to use `--no-interactive` from global args.
Ensures all bootstrap prompts have a default option.
Differential Revision: https://phabricator.services.mozilla.com/D106814
The retry limit doesn't add much value - if user doesn't want
to answer a prompt, they can CTRL-C.
Additionally, `prompt_yesno` doesn't have a retry limit, so
this increases prompt behaviour consistency.
Differential Revision: https://phabricator.services.mozilla.com/D107122
`mach doc mach-telemetry` wasn't run the last time the telemetry
changed. By running it now, the docs are now up-to-date.
Differential Revision: https://phabricator.services.mozilla.com/D106544
Xcode hasn't been necessary to build Firefox for a while. Command line
tools are enough, but if Xcode is available, that's also sufficient.
Mach bootstrap, however, for historic reasons, mostly, ends up wanting
to install both, which is a waste of time and disk space.
Differential Revision: https://phabricator.services.mozilla.com/D107770
* `_format_metadata()` had unused parameters
* `debugger_args` weren't used after being `shellutil.split(...)`
* There was dead python2-specific-code after a `return` statement
Differential Revision: https://phabricator.services.mozilla.com/D106970
- Replace urllib2.urlopen with requests.get. We use requests because
http_download_and_hash, which is also used by __main__ uses it already.
- Convert byte strings to unicode strings.
- Don't wrap sys.stdout, and instead explicitly flush when wanted.
Differential Revision: https://phabricator.services.mozilla.com/D107026
Removes MacPorts as an option for package management on OSX. Changes
most error messages to be Homebrew specific, and rips out the `getattr`
based function resolution used to support both managers.
Differential Revision: https://phabricator.services.mozilla.com/D107277
Upstream wptserve just switched to Python 3 only. That's fine for
web-platform-tests, but it turns out that some marionette harness
tests are also using wptserve and are still on Python 2.
Since fixing marionette harness turns out to be non-trivial and this
blocks other wpt work, this patch does the following:
* Temporarily vendors the last wptserve revision that works with
Python 2 in to testing/web-platform/mozilla/tests/tools/wptserve_py2
* Configures the mach virtualenv to use that copy for Python 2 modules
only.
* Configures the test packaging system to also put that copy in the
common tests zip. Requirements files are updated to use either the
Python 2 version or the Pyhton 3 version as required.
Differential Revision: https://phabricator.services.mozilla.com/D106764
This patch:
-Maintains consistency between the fxa and remote tabs' panels "sync now" buttons
-Removes a fluent string no longer in use
-Restores tooltip functionality
Differential Revision: https://phabricator.services.mozilla.com/D106651
This patch:
-Maintains consistency between the fxa and remote tabs' panels "sync now" buttons
-Removes a fluent string no longer in use
-Restores tooltip functionality
Differential Revision: https://phabricator.services.mozilla.com/D106651
The mozjs_sys crate is actually responsible for all the problems
encountered with the js crate we just removed, but the sm-mozjs-crate
task that builds and runs tests for it doesn't fail because... there is
actually no test to build and run, so all it builds is a static library,
which doesn't expose all the problems that exist.
The content of this crate in mozilla-central is also outdated compared
to servo upstream.
Differential Revision: https://phabricator.services.mozilla.com/D106390
The sm-rust-bindings task that builds and runs tests for it actually only
doesn't fail by three(!) distinct happenstances of chance:
- It somehow doesn't hit a race condition that can happen, but that
attempting to change the build process for jsglue.cpp did,
- The second call to `cargo test` without `--feature debugmozjs` only
works because nothing was cleaned up from the first and the second
build uses artifacts from the first that are otherwise not there if
only running the second.
- The addition of rust code to spidermonkey (the jsrust crate in
js/src/rust) somehow didsn't cause problems in the current
configuration, but in other configurations (which fixing the second
thing above unveils) the build fails because of the multiple
definition of rust_eh_personality between the jsrust staticlib and the
rust stdlib that rustc links to test programs.
On top of all the above, the content of the crate in mozilla-central
has not received updates from upstream servo in years, and doesn't
reflect what servo actually uses.
Differential Revision: https://phabricator.services.mozilla.com/D106389
We always set the app as Android's debug app, which causes Android to wait for
a debugger to connect to the app before allowing it to start. We should clear
that setting once the debugger exits so that other developer activities (such as
manual testing) aren't blocked on the need to connect a debugger.
Differential Revision: https://phabricator.services.mozilla.com/D106807
This patch:
-Maintains consistency between the fxa and remote tabs' panels "sync now" buttons
-Removes a fluent string no longer in use
-Restores tooltip functionality
Differential Revision: https://phabricator.services.mozilla.com/D106651
Now that all the conflicts in python dependencies that made the
new pip resolver fail have been resolved, we no longer need to support
the legacy resolver that was added as a temporary measure.
Depends on D106154
Differential Revision: https://phabricator.services.mozilla.com/D106503
We're not 100% certain if Help is the right spot for this, but we're going
to give it a shot and see. If it turns out it _is_ the right spot, we'll
probably do something a little more self-contained, and less hacky.
I'm leaving the old .properties file just in case we change our mind here.
Yes, we'll want to port to Fluent anyways, but our ultimate choice of where
we put this thing is probably going to dictate where the string lives.
Differential Revision: https://phabricator.services.mozilla.com/D104832
Bug 1476231 actually removed libav, so we don't build it, and don't need
neither the yasm check nor the LIBAV_FFT_ASFLAGS variable.
However, we still have checks, both in moz.build and code, for
MOZ_LIBAV_FFT, so we need to keep that.
Differential Revision: https://phabricator.services.mozilla.com/D105399
* We add new options to the Android variant of `mach run`:
* `--debug`: enables running with a debugger;
* `--debugger`: Allows the user to override the default debugger (`lldb`).
The provided argument must still be `lldb` compatible; this
is for enabling the ability to specify some kind of wrapper
script or other debugger front-end, if desired;
* `--debugger-args`: Additional arguments to pass to the debugger's command line;
* `--no-attach`: Runs the app and prepares the device for debugging, but does
not actually attach any debuggers. The required ports are
printed to the log, and then `mach` exits, thus allowing for
the user to manually connect.
* `--use-existing-process`: This allows the user to attach to an existing
process, instead of killing existing process(es)
and starting from scratch. This is useful for
users who want to attach to an existing process
that is already in a desired state.
When debugging is enabled:
BEFORE the app starts:
* `verify_android_device` will install `lldb-server` if necessary;
* We run `am set-debug-app -w --persistent` to ensure that the app is set as the
device's debug app. Since we pass `-w`, when Android starts the target app, it
will wait for `jdb` to attach before proceeding.
AFTER the app starts:
* We start `lldb-server` and obtain the name of its socket file;
* We obtain the pid of the parent process. Alternatively, if
`--use-existing-process` was specified and there are already extant child
processes, we prompt the user to choose which process to which they would
initially like to attach.
* We forward a local TCP port for `jdb` debugging.
* We run `jdb` in the background to connect to the process and then quit.
This is solely for the purpose of dismissing Android's "waiting for debugger"
dialog.
* In the foreground, we run `lldb`, specifying a set of initial commands that
are required to for symbol resolution and to automatically connect to the
target pid.
Why `lldb`? I chose it for consistency with Android Studio. Somebody else is
welcome to implement `gdb` support if they wish. :-)
Differential Revision: https://phabricator.services.mozilla.com/D94384
NDK 21 includes `lldb-server`, which we need in order to support
`./mach run --debug` with `lldb`.
The Android SDK manager no longer includes a standalone `lldb` package; perhaps
it was deprecated? Anyway, this appears to currently be the best way to get
`lldb-server` into a location that is easy to find during build configuration.
Differential Revision: https://phabricator.services.mozilla.com/D94379
The subprocess.* wrapping in configure alters the environment sent
to the subprocess in two ways:
- variable keys and values are normalized to unicode to make python
happy.
- when no explicit environment is passed, default to the sandbox
environment.
The sandbox environment has one major difference with the original
environment, which is that PYTHONEXECUTABLE is unset, and that's known
to cause problems on mac, which it does when configure executes
`mach artifact toolchain` for --enable-bootstrap.
Differential Revision: https://phabricator.services.mozilla.com/D105265
Indidentally, this fixes the failure code path, because the change in
bug 1692103 was such that the logger output was not captured in `out`.
Differential Revision: https://phabricator.services.mozilla.com/D105390
As far as I can tell, we don't use zstandard from python2. As the last
version supporting python2 is 0.14.1, drop installing the python2
version.
Differential Revision: https://phabricator.services.mozilla.com/D105075
* Puts the docs in order, so that contributors aren't jumping to the
middle of the page to install system tools, then back to the top to
clone Firefox.
* Removes docs on MacPorts since it's being removed in bug 1688263.
* Removes step to manually install brew packages since that happens
automatically in bootstrap now.
* Simplifies mercurial installation docs
* Removes unnecessary mozconfig-tweaking instructions
* Removes almost-always-unnecessary DEFINE and troubleshooting
information.
Differential Revision: https://phabricator.services.mozilla.com/D102973
Bug 1690930 added sysroots that can be bootstrapped. With this change,
we allow --enable-bootstrap=install to pull the right sysroot for the
configured target, and --enable-bootstrap to update it if it was already
there.
Differential Revision: https://phabricator.services.mozilla.com/D104797
The code in MozbuildObject.get_mozconfig_and_target relies on the
configure sandbox to find mozconfig (and target). With
--enable-bootstrap, configure itself ends up calling into taskgraph code
that, eventually ends in in MozbuildObject.get_mozconfig_and_target,
which, because it currently logs to the same place, has at least the
following two effects:
- MOZ_CONFIGURE_TRACE logging breaks. I'm not sure exactly why, but the
log level is lost.
- the output from the configure code that runs for
MozbuildObject.get_mozconfig_and_target ends up mixed with the output
from configure itself, and it appears to the user as if things happened
twice (which it did, actually, but that's not something that should be
shown to the user).
So, we redirect to a separate logger.
Differential Revision: https://phabricator.services.mozilla.com/D104776
This lets Treeherder pick up the line as failure line and show it to Try users
and code sheriffs.
Rule used: "^[A-Za-z.]+Error: "
It also adds an instruction how to fix the reported issue.
Differential Revision: https://phabricator.services.mozilla.com/D104452
Other platforms are going to need more annotations, so this allows us to
update one line while rolling out the feature, rather than updating a
bunch of test expectations. Also should be clearer.
Let me know if there are better ways to do this.
Differential Revision: https://phabricator.services.mozilla.com/D104102
Other platforms are going to need more annotations, so this allows us to
update one line while rolling out the feature, rather than updating a
bunch of test expectations. Also should be clearer.
Let me know if there are better ways to do this.
Differential Revision: https://phabricator.services.mozilla.com/D104102
It's already part of Firefox, and makes the gtest initialization print
an error message because the one already in Firefox can't be
overwritten.
Differential Revision: https://phabricator.services.mozilla.com/D103475
Bug 1553230 made configure automatically get toolchain artifacts in some
cases. The artifacts for clang-cl builds are clang.tar.zst, and extract to
clang/. Configure derives the task name from that knowledge, and fails
to find clang-cl tasks because of that.
For consistency, these tasks should be called clang. They are clang
builds anyways, and like any other clang builds, they also contain
clang-cl.
Differential Revision: https://phabricator.services.mozilla.com/D103150
This allows to filter chrome manifest registration by the current
background task(s, in the future). Filtration behaves just like
filtering by "application":
* filter with `backgroundtask=` means disable for all background
tasks, since no background task will match ""
* filter with `backgroundtask!=` means enable for all background task,
since every background task will not match ""
Differential Revision: https://phabricator.services.mozilla.com/D96482
This adds a --enable-bootstrap build flag that will automatically update
cbindgen, node, clang, sccache, nasm, wine, lucetc, dump_syms, pdbstr,
and winchecksec if they are already installed in ~/.mozbuild.
Eventually, we'll want to allow to install toolchains that weren't
already install, but one step at a time.
This explicitly doesn't cover rustc, which is its own can of worms, or
android-{ndk,sdk}, which are not installed via toolchain artifacts
currently.
Differential Revision: https://phabricator.services.mozilla.com/D101723
Now that we use an external dump_syms, we don't need to build
breakpad's.
This means we also don't need the dump_syms_rust_demangle crate anymore.
Differential Revision: https://phabricator.services.mozilla.com/D101865
These are the minimum changes that we need to make to common build system code
to allow us to generate files during pre-export.
We add a `required_before_export` flag to `GeneratedFile` to indicate when a
particular file must be generated in `pre-export`. We set that flag when there
are `.jinja` input files and we're configured for a GeckoView build, otherwise
it is set to `False`.
Then the recursive `make` backend assigns any `GeneratedFile`s that have
`required_before_export` set to run in the `pre-export` tier.
Differential Revision: https://phabricator.services.mozilla.com/D82576
These are the minimum changes that we need to make to common build system code
to allow us to generate files during pre-export.
We add a `required_before_export` flag to `GeneratedFile` to indicate when a
particular file must be generated in `pre-export`. We set that flag when there
are `.jinja` input files and we're configured for a GeckoView build, otherwise
it is set to `False`.
Then the recursive `make` backend assigns any `GeneratedFile`s that have
`required_before_export` set to run in the `pre-export` tier.
Differential Revision: https://phabricator.services.mozilla.com/D82576
There's a macOS-specific wrinkle for browser/ that populates the
`.app` directory. This makes that happen as part of `mach
package-multi-locale`. It's the equivalent, I suppose, of `mach
android assemble-app` for Desktop.
Differential Revision: https://phabricator.services.mozilla.com/D101502
Since zstandard has native code that must be compiled, and that code
uses Python headers, we should be installing those headers as part
of bootstrap.
Most users will have these packages on their machines through various
other means (notably installing `pip`, ie `sudo apt install python3-pip`),
but since it is possible to avoid a pip installation (for example
by installing Mercurial through `yum` and then running bootstrap
immediately after cloning) we should specify these packages as required
by bootstrap.
Differential Revision: https://phabricator.services.mozilla.com/D101479
The current recommendation fails while waiting on user input. Instead, just
save the script to disk as an intermediate step, then invoke it.
Differential Revision: https://phabricator.services.mozilla.com/D101228
Knowing whether `brew` or `macports` is available isn't necessary
to generate the android mozconfig.
This should fix the generation of android mozconfig when a package
manager isn't available.
Differential Revision: https://phabricator.services.mozilla.com/D99496
pylint_requirements.txt fail to install with the new pip resolver due
to a conflict between astroid and lazy-object-proxy.
Rather than bumping those packages and handling the potential fallout,
the package-upgrade has been deferred and we will use the legacy
resolver in the interrim.
Differential Revision: https://phabricator.services.mozilla.com/D99940
The `AUTOCLOBBER` mozconfig option is reliably causing builds to fail when
a clobber is triggered. When we auto-clobber a build we do so after running
`configure` but before running `make client.mk`. This means we destroy all
the gathered information from the `configure` step in the objdir and then
attempt to run `make` using the previously destroyed information.
This commit moves the call to `_check_clobber` to an earlier stage in the
build process, before `configure` is called, so any clobber that takes place
will happen before setting up the objdir via `configure`.
Since `_check_clobber` is only called once in the codebase, and both cases
are now adding clobber metrics one after another, we remove the metrics
gathering from `_check_clobber` and rely on callers to set metrics instead.
Also clean up some nested `if` statements that can be flattened.
Differential Revision: https://phabricator.services.mozilla.com/D100794
This function has a few code paths and has a slightly confusing return value.
Add a comment describing what it does and what the return value actually
means.
Differential Revision: https://phabricator.services.mozilla.com/D100793
Since the printed value is a `str` anyways, this causes the converted `bytes`
to be printed to the terminal as `b'/path/to/topobjdir'`. Just print the `str`
version to the screen instead.
Differential Revision: https://phabricator.services.mozilla.com/D100792
When mach errors out, an error report is sent to Sentry. This error
report contains information about the state of the interpreter during
the failure, details about the environment, installed packages and more.
Having this information available immediately when attempting to resolve
a bug report is generally desirable, instead of going through a back-and-forth
needinfo tag on Bugzilla or spending time asking the reporter questions on
Matrix.
This commit captures the Sentry ID returned from `sentry_sdk.capture_exception`
and prints it to the screen. If a user adds this line to their bug report (as
the error messages suggest) a build team member can enter this number into
Sentry to identify the exact report and debug the error. At minimum this will
reduce the amount of back-and-forth between the reporter and the assignee
required to resolve a bug. Optimally it should make bugs easier to spot and
reduce the time spent on end user support requests.
To use the Sentry ID to identify information about a specific bug report, the
bug assignee should open the Mozilla Sentry page for the `mach` project and
paste the ID into the search box, which will produce the full stack trace with
all submitted information.
Differential Revision: https://phabricator.services.mozilla.com/D100247
Knowing whether `brew` or `macports` is available isn't necessary
to generate the android mozconfig.
This should fix the generation of android mozconfig when a package
manager isn't available.
Differential Revision: https://phabricator.services.mozilla.com/D99496
pylint_requirements.txt fail to install with the new pip resolver due
to a conflict between astroid and lazy-object-proxy.
Rather than bumping those packages and handling the potential fallout,
the package-upgrade has been deferred and we will use the legacy
resolver in the interrim.
Differential Revision: https://phabricator.services.mozilla.com/D99940
Some distros include flags when they specify the location of a binary,
such as: `XARGS=xargs -r`.
This was confusing in `configure`, since:
* We require that environment variables contain only path
overrides (without flags).
* The error message thrown when configure chokes was unclear: "why
would `$ xargs -r` fail?"
This patch should make our "path-only" requirement more clear.
Differential Revision: https://phabricator.services.mozilla.com/D100044
At the moment installing Python packages with native code is done by calling
`pip install <package>` and does not enforce any SHA hash for installed
dependencies, nor does it enforce a specific version to be installed.
This commit adds `requirements.in` and `requirements.txt` files for native
packages and changes these packages to be installed by running `pip install`
and passing the requirements file for the package. This allows us to pin the
SHA of the various dependencies. The `.txt` files are generated using
`pip-compile`.
We also add the new requirements files to the sparse profile for `mach`.
Differential Revision: https://phabricator.services.mozilla.com/D99912