Граф коммитов

6435 Коммитов

Автор SHA1 Сообщение Дата
Mitchell Hentges 4bdf055603 Bug 1661298: Log if artifact fails and Taskcluster URL overridden. r=froydnj
It can be tricky to find out why artifact-lookup fails, especially since
it can be implicitly affected by environment variables.
Log if a known troublemaker (TASKCLUSTER_ROOT_URL) is set when an
artifact-lookup fails.

Differential Revision: https://phabricator.services.mozilla.com/D88304
2020-08-26 17:33:31 +00:00
Tarek Ziadé bbb7d4b5d5 Bug 1659411 - add missing dep r=sparky
Differential Revision: https://phabricator.services.mozilla.com/D88295
2020-08-26 15:47:02 +00:00
Ricky Stewart 6e1d74e077 Bug 1624667 - Allow running `mach jit-test` if tests are disabled r=mhentges,froydnj
This works fine, so no reason not to allow it.

Differential Revision: https://phabricator.services.mozilla.com/D86398
2020-08-26 14:45:52 +00:00
Ricky Stewart 93f6564a24 Bug 1660124 - Don't trample existing `mach` `virtualenv`s if they're already up-to-date r=mhentges,ahal
On some platforms re-creating the `virtualenv`s can be very time-consuming so we'd like to avoid deleting these `virtualenv`s unnecessarily.

Preserve the existing behavior behind a `-f` flag in case unconditionally wiping the `virtualenv`s is what's needed for any reason.

Differential Revision: https://phabricator.services.mozilla.com/D87668
2020-08-25 21:21:19 +00:00
june wilde fcec357fc8 Bug 1661021 - add timestamp to mach vendor --check-for-update output r=tjr
Differential Revision: https://phabricator.services.mozilla.com/D88148
2020-08-25 14:38:41 +00:00
Mike Hommey 922b27a089 Bug 1660880 - Support topobjdir-relative generated files in the faster make backend. r=froydnj
Bug 1659906 removed `../`s from some generated files paths which turned
up not being supported by the faster make backend. Instead of returning
to those relative paths, just support the topobjdir-relative paths
correctly. The new code is derived from the equivalent code in the
recursive make backend.

Differential Revision: https://phabricator.services.mozilla.com/D88097
2020-08-25 01:38:33 +00:00
Ricky Stewart a292021326 Bug 1660566 - Warn in `artifacts.py` if no git revisions are found r=mhentges,froydnj
This generally happens because people cleverly create custom forks of `mozilla-central` that don't have `git-cinnabar` metadata. This is ALWAYS broken in for artifact builds, but people generally don't know that and the error message isn't informative. Instead, identify when this happens as it happens and suggest an immediate, working alternative.

Differential Revision: https://phabricator.services.mozilla.com/D87923
2020-08-24 15:47:56 +00:00
Nick Alexander 9c623dbc86 Bug 1658040 - Part 2: Deny `mach package-multi-locale ...` when it's going to fail. r=froydnj
Depends on D87331

Differential Revision: https://phabricator.services.mozilla.com/D87332
2020-08-18 17:07:47 +00:00
Nick Alexander a1aff5b241 Bug 1658040 - Part 1: Fix local `mach package-multi-locale ...` locally. r=snorp
This succeeded in automation because we [check for `MOZILLA_OFFICIAL`
before invoking `mach` from within
Gradle](https://searchfox.org/mozilla-central/rev/8df04ff073d0fa70f692935c5dcddc0e23cb0117/build.gradle#110-146).
Locally, the check for `AB_CD=multi` in the environment used to catch
this but it's no longer used.  Rather than continue confusion with
`AB_CD=multi make ...` versus `make AB_CD=multi ...`, we'll use the
Gradle-specific environment variable.

Differential Revision: https://phabricator.services.mozilla.com/D87331
2020-08-18 18:13:26 +00:00
Ricky Stewart 64d5f3b565 Bug 1660105 - Allow installing `glean_sdk` to fail in `create-mach-environment`. r=mhentges,froydnj
`glean_sdk` can't currently be installed on Apple Silicon or OpenBSD since Glean can't be built from source. While that issue is being resolved (see bug 1660120), allow this installation to fail.

Differential Revision: https://phabricator.services.mozilla.com/D87667
2020-08-20 16:46:59 +00:00
Mike Hommey 9c86ddc0da Bug 1659411 - Default virtualenv_name to None in MachCommandBase. r=firefox-build-system-reviewers,rstewart
Bug #985141 added this argument without changing all the callers.
Instead of fixing each caller individually, just allow a value not to be
passed in. This is what the underlying MozbuildObject class does
anyways.

Differential Revision: https://phabricator.services.mozilla.com/D87386
2020-08-18 12:41:07 +00:00
Mark Hammond b0ffe6a42b Bug 1659615 - handle the fact Windows Defender might have exclusions on different disks. r=mhentges,froydnj
Differential Revision: https://phabricator.services.mozilla.com/D87382
2020-08-18 11:28:50 +00:00
Ricky Stewart 1218762d08 Bug 1659575 - Delete `mach python-safety` r=ahal
There are zero uses of this `mach` command over the past 90 days according to our telemetry. There are no external references to `mach python-safety` in-tree, and indeed if you track the history of the originating bug 1468394, it appears that once the `mach` command was created, none of the follow-up work that was discussed (i.e. running this in CI and triaging failures to appropriate owners) was done over the following 2 years.

If this ever does appear to be useful in the future, we can just resurrect this code from source control.

Differential Revision: https://phabricator.services.mozilla.com/D87351
2020-08-18 14:18:24 +00:00
Jonathan Kew 65ccd8816d Bug 1659685 - Quote parameters passed to mozconfig_loader, in case of spaces in paths. r=rstewart
Differential Revision: https://phabricator.services.mozilla.com/D87427
2020-08-18 13:28:32 +00:00
Ricky Stewart 9ad232391e Bug 1659602 - Fix type error in `mozversioncontrol` for Windows+`hg`+Python 3 r=dmajor
These arguments will get converted to `bytes` later on if necessary; if we don't defer to `hglib`, then we need proper strings.

Differential Revision: https://phabricator.services.mozilla.com/D87440
2020-08-18 12:51:49 +00:00
Ricky Stewart bbfa258584 Bug 1656993: Create and require by default global `virtualenv`s in `~/.mozbuild` for `mach` r=mhentges,ahal
In two different places we've been encountering issues regarding 1) how we configure the system Python environment and 2) how the system Python environment relates to the `virtualenv`s that we use for building, testing, and other dev tasks. Specifically:

1. With the push to use `glean` for telemetry in `mach`, we are requiring (or rather, strongly encouraging) the `glean_sdk` Python package to be installed with bug 1651424. `mach bootstrap` upgrades the library using your system Python 3 in bug 1654607. We can't vendor it due to the package containing native code. Since we generally vendor all code required for `mach` to function, requiring that the system Python be configured with a certain version of `glean` is an unfortunate change.

2. The build uses the vendored `glean_parser` for a number of build tasks. Since the vendored `glean_parser` conflicts with the globally-installed `glean_sdk` package, we had to add special ad-hoc handling to allow us to circumvent this conflict in bug 1655781.

3. We begin to rely more and more on the `zstandard` package during build tasks, this package again being one that we can't vendor due to containing native code. Bug 1654994 contained more ad-hoc code which subprocesses out from the build system's `virtualenv` to the SYSTEM `python3` binary, assuming that the system `python3` has `zstandard` installed.

As we rely more on `glean_sdk`, `zstandard`, and other packages that are not vendorable, we need to settle on a standard model for how `mach`, the build process, and other `mach` commands that may make their own `virtualenv`s work in the presence of unvendorable packages.

With that in mind, this patch does all the following:

1. Separate out the `mach` `virtualenv_packages` from the in-build `virtualenv_packages`. Refactor the common stuff into `common_virtualenv_packages.txt`. Add functionality to the `virtualenv_packages` manifest parsing to allow the build `virtualenv` to "inherit" from the parent by pointing to the parent's `site-packages`. The `in-virtualenv` feature from bug 1655781 is no longer necessary, so delete it.

2. Add code to `bootstrap`, as well as a new `mach` command `create-mach-environment` to create `virtualenv`s in `~/.mozbuild`.

3. Add code to `mach` to dispatch either to the in-`~/.mozbuild` `virtualenv`s (or to the system Python 3 for commands which cannot run in the `virtualenv`s, namely `bootstrap` and `create-mach-environment`).

4. Remove the "add global argument" feature from `mach`. It isn't used and conflicts with (3).

5. Remove the `--print-command` feature from `mach` which is obsoleted by these changes.

This has the effect of allowing us to install packages that cannot be vendored into a "common" place (namely the global `~/.mozbuild` `virtualenv`s) and use those from the build without requiring us to hit the network. Miscellaneous implementation notes:

1. We allow users to force running `mach` with the system Python if they like. For now it doesn't make any sense to require 100% of people to create these `virtualenv`s when they're allowed to continue on with the old behavior if they like. We also skip this in CI.

2. We needed to duplicate the global-argument logic into the `mach` script to allow for the dispatch behavior. This is something we avoided with the Python 2 -> Python 3 migration with the `--print-command` feature, justifying its use by saying it was only temporarily required until all `mach` commands were running with Python 3. With this change, we'll need to be able to determine the `mach` command from the shell script for the forseeable future, and committing to this forever with the cost that `--print-command` incurs (namely `mach` startup time, an additional .4s on my machine) didn't seem worth it to me. It's not a ton of duplicated code.

Differential Revision: https://phabricator.services.mozilla.com/D85916
2020-08-17 17:21:02 +00:00
Ricky Stewart 9719a9a5a9 Bug 1636797 - In `hash.py`, enumerate files from the VCS rather than searching the filesystem directly r=ahal
This resolves a long-standing issue in development where `mach artifact` (and therefore `mach bootstrap`) would fail unpredictably if you had dirty, but ignored, files in your checkout. Resolving this problem often required unwieldy `hg purge`/`git ignore` incantations that are easy to get wrong.

This patch addresses the problem by doing what we "should" have been doing all along, and consulting the VCS to list tracked files rather than listing EVERY file on disk and applying heuristics to determine whether they should be included in the hash.

Differential Revision: https://phabricator.services.mozilla.com/D86780
2020-08-17 15:19:34 +00:00
Ricky Stewart 7301a3d06b Bug 1659154 - Bad call to __init__ of Build object in mach ide after bug 985141 r=froydnj,dmajor
Differential Revision: https://phabricator.services.mozilla.com/D87103
2020-08-14 17:39:39 +00:00
Ricky Stewart 2eae41178a Bug 985141 - Allow mach commands to specify what virtualenv they should use. r=mhentges,dmajor
Now you can pass the `virtualenv_name` kwarg to the `Command` decorator which will configure the `_virtualenv_manager` accordingly.

Differential Revision: https://phabricator.services.mozilla.com/D86256
2020-08-13 22:19:44 +00:00
Csoregi Natalia 8027f811c2 Backed out changeset 330b617b585c (bug 985141) for bustage and test info failures due to missing argument. CLOSED TREE 2020-08-14 01:09:29 +03:00
Ricky Stewart c3dc1fbe77 Bug 1658651 - When determining whether `virtualenv`s are up to date, don't take file size into account r=mhentges,ahal
This check is unsound; `virtualenv` binaries are apparently not guaranteed to have the same file size as the `python` binaries used to create those `virtualenv`s, at least not with our current vendored version of the `virtualenv` library on macOS. This is trivially reproducible on my own Macbook:

```
rickystewart-a5lvdq:mozilla-unified rickystewart$ rm -rf obj-x86_64-apple-darwin19.5.0/
rickystewart-a5lvdq:mozilla-unified rickystewart$ ./mach configure
...
rickystewart-a5lvdq:mozilla-unified rickystewart$ python3 -c 'import os; print(os.path.getsize("obj-x86_64-apple-darwin19.5.0/_virtualenvs/init_py3/bin/python"))'
16644  # <- ACTUAL VIRTUALENV SIZE
rickystewart-a5lvdq:mozilla-unified rickystewart$ python3 -c 'import os; print(os.path.getsize("/usr/local/opt/python/bin/python3.7"))'
17704  # <- SIZE OF THE PYTHON USED TO CREATE THE VIRTUALENV
```

Concretely, this was causing unit tests to be very aggressive about deleting the parent `init_py3` `virtualenv` repeatedly in unit tests, resulting in failures. The removal of this check fixes the issue.

Differential Revision: https://phabricator.services.mozilla.com/D86872
2020-08-13 19:29:54 +00:00
Ricky Stewart fcb0052827 Bug 985141 - Allow mach commands to specify what virtualenv they should use. r=mhentges,dmajor
Now you can pass the `virtualenv_name` kwarg to the `Command` decorator which will configure the `_virtualenv_manager` accordingly.

Differential Revision: https://phabricator.services.mozilla.com/D86256
2020-08-13 18:26:07 +00:00
Mike Hommey d736c1381f Bug 1658853 - Make GnMozbuildWriter output more deterministic. r=froydnj
Differential Revision: https://phabricator.services.mozilla.com/D86887
2020-08-13 13:20:04 +00:00
Mitchell Hentges 2e29bfb1f0 Bug 1658486: Don't re-install pipenv if "pipenv.exe" exists on Windows r=firefox-build-system-reviewers,rstewart
To check if pipenv is installed, the existing logic looked to see if the "pipenv" binary existed. However, on Windows, this binary is named "pipenv.exe".

Differential Revision: https://phabricator.services.mozilla.com/D86680
2020-08-11 18:43:51 +00:00
Mitchell Hentges 031dcb9d05 Bug 1657717: pipenv should use same version of python as currently active, if possible r=firefox-build-system-reviewers,rstewart
Pipenv enthusiastically checks pyenv for possible Python installations, and it seems to always want to use the most
up-to-date version possible. This can lead to issues if the version of python used for the regular venvs is older
than another version of python that exists on the machine (such as if the system python is 3.8.2, but pyenv has
3.8.3 installed).

This patch addresses this by scoping pipenv to match the currently-running version of python (if major versions line
up).

Differential Revision: https://phabricator.services.mozilla.com/D86448
2020-08-12 20:14:37 +00:00
Ricky Stewart f6051c3aa8 Bug 1658626 - When listing files from git in `mozversioncontrol`, ignore empty paths r=froydnj,mhentges
Without this patch, the last "path" in this list will always be the empty string due to how the `-z` option to `git` works. This mirrors what we already do in the `get_files_in_working_directory` implementation for `hg`.

Differential Revision: https://phabricator.services.mozilla.com/D86752
2020-08-12 00:10:43 +00:00
Ben Hearsum 9ece148d45 Bug 1607619: only run rusttests when rust files are modified. r=ahal
Differential Revision: https://phabricator.services.mozilla.com/D86434
2020-08-12 14:36:12 +00:00
Mitchell Hentges 710be25312 Bug 1655361: Prints a warning if Windows Defender is watching the Firefox checkout directory r=firefox-build-system-reviewers,rstewart
There's a Windows Defender CLI (`Get-MpComputerStatus`) available, but MozillaBuild
can't (easily) access PowerShell. So, instead, we find Windows Defender status and path exclusions
by checking the registry at HKLM\SOFTWARE\Microsoft\Windows Defender.

Determining if Windows Defender is "on" or not is a surprisingly vague task.
Not only does it encompass a lot of components (of which Ricky and I believe "Real-time protection"
is the part slowing down the build), but there's (at least?) two different kinds of "disabled"
states that it can be in:
* If "disabled" via Settings, it will turn itself back on after a reboot
* If an antivirus is installed, it will turn itself off permanently

Unfortunately, disabling "Real-time protection" in Settings doesn't affect any registry keys, but
I'm opting to ignore this because I doubt many users are manually disabling this protection before
each build. The repercussion of this is that users may be incorrectly warned "your Firefox
directory isn't excluded from Windows Defender!".

Focusing on the antivirus use case and querying the registry I found that:
* Querying the `IsServiceRunning` value wasn't sufficient because new Windows installations didn't have that key
* Querying the `DisableRealtimeMonitoring` value was inconsistent - it can be missing, and not always because an antivirus removed it
* Querying the `DisableAntiVirus` value doesn't _sound_ accurate (we care about Real-time Protection), but it's consistently "off" for non-AV machines, and "on" for machines with an AV installed. So, this is our winner!

TL;DR: there may be some "false positive" warnings about excluding the Firefox srcdir, but they're
accurate for my test cases and workaround-able (just add the exclusion to Windows Defender).

Also, this patch updates our Windows Defender docs to make them:
* More easily link-able
* Have direct advice to resolve the issue

Differential Revision: https://phabricator.services.mozilla.com/D85952
2020-08-11 20:59:40 +00:00
Ricky Stewart e6d1d2a4dd Bug 1656614 - Delete existing `virtualenv` before re-creating it r=mhentges,ahal
I wrote this patch because I noticed that the `.pth` files in my `objdir` `virtualenv`s were extremely repetitive, containing multiple references to the same directories. This happens because we [append](https://searchfox.org/mozilla-central/rev/03794edd6edcc3fc1e222de966cb27256ce08998/python/mozbuild/mozbuild/virtualenv.py#366) to the in-`virtualenv` `.pth` files when calling `populate()`, but we don't ever clean up the old ones, meaning that whenever we determine that the `virtualenv`s are out of date and need to be recreated, we actually leave A LOT of state lying around on-disk that is going to go on to impact further uses of the `virtualenv`. Concretely, how this manifested is that when I erroneously removed an entry from `virtualenv_packages.txt`, the build actually succeeded because that entry was still in the `.pth` file in the `virtualenv`; instead of "creating" a new `virtualenv` with the correct `.pth` files, it just appended the new `.pth` data to the old, stale data.

I've chosen to address this by completely deleting the entire `virtualenv` when we try to re-create it. Another way you might solve this problem is by doing a `find $VIRTUALENV -name '*.pth' | xargs rm` before doing the `virtualenv` re-creation, but I'm suggesting we do it this way because we have had a long history of difficulty with `virtualenv` persistence. Bug 1628498 is an obvious example; note that we would never have encountered that bug if we always unconditionally deleted the `virtualenv` before creating a new one, as in this patch. A patch that is laser-targeted at handling the issue with `.pth` files might be fine for now but this is more foolproof and future-proof.

Differential Revision: https://phabricator.services.mozilla.com/D85636
2020-08-07 15:35:27 +00:00
Tarek Ziadé 2909efb662 Bug 1650363 - Performance testing of HTTP3 in Firefox using only xpcshell tests r=michal,sparky
Differential Revision: https://phabricator.services.mozilla.com/D84021
2020-08-10 06:26:39 +00:00
Mike Hommey c9ae9101aa Bug 1651680 - Replace PKG_SKIP_STRIP with PKG_STRIP. r=nalexander
Differential Revision: https://phabricator.services.mozilla.com/D86648
2020-08-11 02:44:17 +00:00
Razvan Maries 6d82f7f1a0 Backed out 2 changesets (bug 1651680) for L10n bustages. CLOSED TREE
Backed out changeset 09a5f4dcd92a (bug 1651680)
Backed out changeset 13a881966dda (bug 1651680)
2020-08-11 05:40:47 +03:00
Mike Hommey 6714f1ec83 Bug 1651680 - Replace PKG_SKIP_STRIP with PKG_STRIP. r=nalexander
Differential Revision: https://phabricator.services.mozilla.com/D86648
2020-08-11 00:07:02 +00:00
Gregory Mierzwinski f87d8a4859 Bug 1657898 - Fix fenix artifact paths. r=jlorenzo,tarek
Differential Revision: https://phabricator.services.mozilla.com/D86351
2020-08-08 12:16:49 +00:00
Tarek Ziadé abb4969c3e Bug 1656235 - support visual metrics r=acreskey,sparky
Differential Revision: https://phabricator.services.mozilla.com/D85561
2020-08-08 12:17:48 +00:00
Tarek Ziadé 5a38472369 Bug 1651818 - fixed logger when runner is used directly r=acreskey
Differential Revision: https://phabricator.services.mozilla.com/D86305
2020-08-08 12:17:48 +00:00
Mitchell Hentges 7759de2362 Bug 1657719: Don't create init_py3_py3 venv in python-test r=firefox-build-system-reviewers,rstewart
To ensure that a python 3 virtualenv exists for tests, python-test will ensure that it's created (regardless of which version of python is currently in use).
However, the existing logic was incorrectly creating an "extra py3 virtualenv", even if currently running python 3 and having already created a "first" py3 virtualenv

Differential Revision: https://phabricator.services.mozilla.com/D86269
2020-08-07 22:11:41 +00:00
Ricky Stewart 31755b431d Bug 1657650 - Require that Mach command providers subclass MachCommandBase. r=remote-protocol-reviewers,marionette-reviewers,maja_zf,mhentges,froydnj
Today we don't require that `mach` `CommandProvider`s subclass from any particular parent class and we're very lax about the requirements they must meet. While that's convenient in certain circumstances, it has some unfortunate implications for feature development.

Today the only requirements that we have for `CommandProvider`s are that they have an `__init__()` method that takes either 1 or 2 arguments, the second of which must be called `context` and is populated with the `mach` `CommandContext`. Again, while this flexibility is occasionally convenient, it is limiting. As we add features to `mach`, having a better idea what the shape of our `CommandProvider`s are and how we can instantiate them and use them is increasingly important, and this gives us additional control when having `mach` configure `CommandProvider`s based on data that is only available at the `mach` level. In particular, we plan to leverage this in bugs 985141 and 1654074.

Here we add validation to the `CommandProvider` decorator to ensure all classes inherit from `MachCommandBase`, update all `CommandProvider`s in-tree to inherit from `MachCommandBase`, and update source and test code accordingly.

Follow-up work: we now require (de facto) that the `context` be populated with a `topdir` attribute by the `populate_context_handler` function, since instantiating the `MachCommandBase` requires a `topdir` be provided. This is fine for now in the interest of keeping this patch reasonably sized, but some additional refactoring could make this cleaner.

Differential Revision: https://phabricator.services.mozilla.com/D86255
2020-08-07 18:24:59 +00:00
Mitchell Hentges 7efb4ba7b0 Bug 1657677: Resolves FileNotFoundError when running python-test from PyCharm r=firefox-build-system-reviewers,rstewart
When subprocessing to get the venv from pipenv with PyCharm, the returned path has a command sequence to reset the terminal color.
This command sequence is unexpected, and causes the returned path to be incorrect.

The root issue is that colorama mistakenly believes that, if it's ever running underneath PyCharm, it's probably attached to a tty.
However, this assumption isn't true if PyCharm is running a script which subprocesses out a colorama-enabled script.
This issue has already been reported here: https://github.com/tartley/colorama/issues/263

To workaround this issue, we clear the "PYCHARM_HOSTED" environment variable when we invoke pipenv, which works around the colorama logic.

Differential Revision: https://phabricator.services.mozilla.com/D86242
2020-08-06 20:10:56 +00:00
Ricky Stewart 3ddb065650 Bug 1656611 - Remove `objdir` support from `virtualenv_packages.txt` handling r=mhentges,froydnj
I noticed that the `objdir:build` entry in `build/virtualenv_packages.txt` entry was apparently unused. This originates from bug 841713, seven years ago, when the `objdir` handling was introduced. Today, this doesn't appear to be serving a purpose. There is no Python library in my `$objdir/build` directory; nor can I see anything in `build/moz.build` or any related files suggesting one could ever appear. I can only assume this feature has outlived its usefulness, so delete it and the relevant in-tree support.

This necessitates slightly changing the signature and implementation of the `activate_pipenv()` method; also update all callers.

Differential Revision: https://phabricator.services.mozilla.com/D85635
2020-08-07 16:03:36 +00:00
Bogdan Tara 57ed67928b Backed out changeset 760fc38c793b (bug 1656611) for doc failure complaining about virtualenv_root CLOSED TREE 2020-08-07 18:57:56 +03:00
Ricky Stewart 41e473f17b Bug 1656611 - Remove `objdir` support from `virtualenv_packages.txt` handling r=mhentges,froydnj
I noticed that the `objdir:build` entry in `build/virtualenv_packages.txt` entry was apparently unused. This originates from bug 841713, seven years ago, when the `objdir` handling was introduced. Today, this doesn't appear to be serving a purpose. There is no Python library in my `$objdir/build` directory; nor can I see anything in `build/moz.build` or any related files suggesting one could ever appear. I can only assume this feature has outlived its usefulness, so delete it and the relevant in-tree support.

This necessitates slightly changing the signature and implementation of the `activate_pipenv()` method; also update all callers.

Differential Revision: https://phabricator.services.mozilla.com/D85635
2020-08-06 18:21:35 +00:00
Hamzah 8a86250605 Bug 1408051 - Remove mozharness' copy of virtualenv and use the one under /third_party/python instead r=ahal,rail
Differential Revision: https://phabricator.services.mozilla.com/D83181
2020-08-05 19:52:24 +00:00
Andrew Halberstadt bc270b20f5 Bug 1657604 - [mozlint] Support multiple linters per yaml file, r=linter-reviewers,sylvestre
Differential Revision: https://phabricator.services.mozilla.com/D86206
2020-08-07 07:12:41 +00:00
Mike Hommey 5a5a1edd0e Bug 1656141 - Remove RESFILE. r=firefox-build-system-reviewers,rstewart
The resource file is always generated so being able to configure its name
is not useful. On the other hand, the way things are currently implemented,
the lack of RESFILE also makes RCFILE ignored, which we fix at the same
time.

And remove a spurious RESFILE in widget/windows/moz.build, where no binary
is produced, which means RESFILE had no meaning.

Differential Revision: https://phabricator.services.mozilla.com/D86154
2020-08-06 22:51:59 +00:00
Mihai Alexandru Michis c4e805857f Backed out 9 changesets (bug 1656141) for causing multiple failures.
CLOSED TREE

Backed out changeset 9033b0400339 (bug 1656141)
Backed out changeset e43dd57dc61a (bug 1656141)
Backed out changeset eb450457a9b7 (bug 1656141)
Backed out changeset 194a994cf9c9 (bug 1656141)
Backed out changeset ce6831acb5e3 (bug 1656141)
Backed out changeset 29653ea85d49 (bug 1656141)
Backed out changeset 704f28486bda (bug 1656141)
Backed out changeset de8899453150 (bug 1656141)
Backed out changeset 4b133eda46aa (bug 1656141)
2020-08-07 01:48:45 +03:00
Mike Hommey afe0ac845a Bug 1656141 - Remove RESFILE. r=firefox-build-system-reviewers,rstewart
The resource file is always generated so being able to configure its name
is not useful. On the other hand, the way things are currently implemented,
the lack of RESFILE also makes RCFILE ignored, which we fix at the same
time.

And remove a spurious RESFILE in widget/windows/moz.build, where no binary
is produced, which means RESFILE had no meaning.

Differential Revision: https://phabricator.services.mozilla.com/D86154
2020-08-06 15:59:02 +00:00
Dave Townsend 2e11494093 Bug 1653384: mach run should set MOZ_FORCE_DISABLE_E10S to the current application version. r=froydnj
Differential Revision: https://phabricator.services.mozilla.com/D86240
2020-08-06 18:19:19 +00:00
Andi-Bogdan Postelnicu 992408f681 Bug 1627484 - clang cannot understanding the mixing of double and single quotes. r=botond
Differential Revision: https://phabricator.services.mozilla.com/D70300
2020-08-06 07:54:09 +00:00
Andi-Bogdan Postelnicu 1ead43f591 Bug 1657299 - First step of refactor `static-analysis` integration, making it more modular. r=marco
Add a modular approach for the integration of `static-analysis` module in order
to be able to share components of it with other modules, like the integration of
`clangd` in `vscode` where we need to have access to the configuration of `clang-tidy`
in order to have `in-ide` `static-analysis` messages.
In this initial step we make a separate module for the clang-tidy configuration.

Differential Revision: https://phabricator.services.mozilla.com/D85979
2020-08-06 06:51:16 +00:00
Andi-Bogdan Postelnicu 9e73e83509 Bug 1657301 - Reformat `static-analysis` with `black`. r=sylvestre
# ignore-this-changeset

Differential Revision: https://phabricator.services.mozilla.com/D85985
2020-08-05 13:32:46 +00:00
Andi-Bogdan Postelnicu 60cc6f8d69 Bug 1656740 - Integrate `clangd` in `vscode` for C++ language support. r=froydnj
In order to have a cross platform ide for C++ language support we've added `clangd`
extenssion and artifact part of `vscode` suite.
To generate the configuration you simply run:
`./mach ide vscode `.

Differential Revision: https://phabricator.services.mozilla.com/D85416
2020-08-06 06:25:17 +00:00
Mike Hommey 7347ccb900 Bug 1657446 - Fix co_lnotab for large offsets in Lint._raise_from. r=nalexander
Contrary to python2, python3 considers the values in cl_lnotab to be signed
integers, so with python3, offsets larger than 127 would be encoded in a way
that would make them wrong, or worse, negative.

Differential Revision: https://phabricator.services.mozilla.com/D86122
2020-08-06 04:07:14 +00:00
Mike Hommey d0d43d8f9b Bug 1657446 - Fix noise when running mozbuild/test/configure/lint.py with unittest.main. r=nalexander
Namely:
  build/moz.configure/old.configure:21: DeprecationWarning: invalid escape sequence \s
  build/moz.configure/old.configure:26: DeprecationWarning: invalid escape sequence \s
  python/mozbuild/mozbuild/configure/__init__.py:433: ResourceWarning: unclosed file <_io.BufferedReader...>
  python/mozbuild/mozbuild/configure/lint.py:120: DeprecationWarning: inspect.getargspec() is deprecated since Python 3.0, use inspect.signature() or inspect.getfullargspec()

Differential Revision: https://phabricator.services.mozilla.com/D86121
2020-08-06 03:19:08 +00:00
Jim Porter 8e234c7598 Bug 1537763 - [mozproxy] Add a record mode; r=tarek
Differential Revision: https://phabricator.services.mozilla.com/D84399
2020-08-05 19:43:55 +00:00
Tom Ritter 66394143e9 Bug 1601690 - Remove gcc as a supported compiler for Windows r=froydnj
Differential Revision: https://phabricator.services.mozilla.com/D56046
2020-08-05 18:55:13 +00:00
Tom Prince 4585d70c82 Bug 1657217: Remove unused test binaries; r=firefox-build-system-reviewers,glandium
elfhack is unneeded after Bug 1645842  and filedid after Bug 1619840.

Differential Revision: https://phabricator.services.mozilla.com/D85924
2020-08-05 03:59:51 +00:00
Cosmin Sabou 12db94501a Backed out changeset 3dba9cbe665b (bug 1657217) for mass failures on windows. CLOSED TREE 2020-08-05 04:59:20 +03:00
Bogdan Tara abfa3262b8 Backed out changeset 1ccda1d0c3ea (bug 1650363) for causing a serious regression in packaging time 2020-08-05 04:23:20 +03:00
Tom Prince c8dfe8b2b0 Bug 1657217: Remove unused test binaries; r=firefox-build-system-reviewers,glandium
`elfhack` is unneeded after Bug 1645842 and `fileid` after Bug 1619840.

Differential Revision: https://phabricator.services.mozilla.com/D85924
2020-08-05 01:09:27 +00:00
Ricky Stewart 5bca67c689 Bug 1657198 - Delete broken tests in `test_telemetry.py` r=dmajor
These tests depend on the `mach uuid` command which was deleted with bug 1639509, and now that `mach uuid` is gone it's broken unconditionally. We could replace the reference to `uuid` with a new no-op `mach` command, but we're in the process of replacing our telemetry code with use of the `glean` API; and the new telemetry code won't have the same semantics (namely, we are unlikely to want to continue to guarantee that sub-`mach` invocations aren't covered by telemetry), so this test might as well just be deleted now.

Differential Revision: https://phabricator.services.mozilla.com/D85911
2020-08-04 20:03:01 +00:00
Hamzah b7e6b41fc5 Bug 1594914 - Convert mach python-test to run with python 3 r=gbrown
Differential Revision: https://phabricator.services.mozilla.com/D77872
2020-08-04 17:40:58 +00:00
Yue Zhang cb36d08ccc Bug 1630611 - Remove the nested helper function in the flat() function. r=sparky
Differential Revision: https://phabricator.services.mozilla.com/D85677
2020-08-04 17:15:40 +00:00
Sylvestre Ledru d61fa8949f Bug 1656764 - Relax the version check for clang-format r=andi DONTBUILD
So that it works when CLANG_VENDOR is set

Differential Revision: https://phabricator.services.mozilla.com/D85687
2020-08-03 12:41:15 +00:00
Tarek Ziadé 03cbd4d536 Bug 1650363 - Performance testing of HTTP3 in Firefox using only xpcshell tests r=sparky
Differential Revision: https://phabricator.services.mozilla.com/D84021
2020-08-02 17:28:57 +00:00
Gregory Mierzwinski a4e1ebe204 Bug 1656540 - Allow specifying perfherder settings on a per-metric basis. r=tarek
This patch fixes an issue where the metric settings were not being used because they don't use the test name. It also handles some changes (from a bad copy-paste) that didn't make it into the last live-site patch series.

Differential Revision: https://phabricator.services.mozilla.com/D85609
2020-08-01 16:13:28 +00:00
Mike Hommey 38cc2e0609 Bug 1654994 - Switch full symbols archive to tar.zst. r=firefox-build-system-reviewers,rstewart,tomprince
Differential Revision: https://phabricator.services.mozilla.com/D85252
2020-07-31 20:27:54 +00:00
Mike Hommey 36a8aaf1b4 Bug 1654994 - Support using .tar.zst symbol archives in mach artifact. r=nalexander,firefox-build-system-reviewers,rstewart
Differential Revision: https://phabricator.services.mozilla.com/D85249
2020-07-31 10:01:52 +00:00
Mike Hommey 557b6e2811 Bug 1654994 - Support creating .tar.zst archives instead of .zip for symbol archives. r=firefox-build-system-reviewers,rstewart,mhentges
Differential Revision: https://phabricator.services.mozilla.com/D85248
2020-08-01 05:33:05 +00:00
Mitchell Hentges d0ec90c4d5 Bug 1654074: Make @CommandProvider validation more specific r=firefox-build-system-reviewers,rstewart
@CommandProvider does parameter validation and collects information (such
as "pass_context") that will be needed by Registrar.

However, rather than just checking parameter length, we can make it more
specific and assert that the specific expected parameter ("context") exists.

Differential Revision: https://phabricator.services.mozilla.com/D85482
2020-07-30 22:25:05 +00:00
Yue Zhang 210c0b2307 Bug 1635930 - Activate a layer when its options are used. r=sparky,tarek
Differential Revision: https://phabricator.services.mozilla.com/D83551
2020-07-31 17:23:51 +00:00
Gregory Mierzwinski ee2aa57e5c Bug 1655031 - Fix how mozperftest metrics are parsed. r=tarek
Differential Revision: https://phabricator.services.mozilla.com/D85318
2020-07-31 15:33:12 +00:00
Gregory Mierzwinski d93acb6b75 Bug 1655032 - Add splitting functionality to metrics module. r=tarek
This patch adds the `--<LAYER>-split-by` option to the metric layers. It allows users to split the data they obtain using a given data field name. For instance, if `browserScripts.pageinfo.url` is provided, then the data will be split based on the unique URLs that are found.

Differential Revision: https://phabricator.services.mozilla.com/D84822
2020-07-30 23:12:28 +00:00
Dorel Luca 0226b8feef Backed out 6 changesets (bug 1654994) for build bustage on fuzzy builds. DONTBUILD
Backed out changeset cd3e1074b785 (bug 1654994)
Backed out changeset 45ee50e76d61 (bug 1654994)
Backed out changeset 77f1b667baec (bug 1654994)
Backed out changeset 9fca8b0bf8c1 (bug 1654994)
Backed out changeset 7131943d8db9 (bug 1654994)
Backed out changeset 435b90f6ca70 (bug 1654994)
2020-07-31 12:36:14 +03:00
Mike Hommey 311e167747 Bug 1654994 - Switch full symbols archive to tar.zst. r=firefox-build-system-reviewers,rstewart,tomprince
Differential Revision: https://phabricator.services.mozilla.com/D85252
2020-07-30 22:21:28 +00:00
Mike Hommey 1bf0ac0aca Bug 1654994 - Support using .tar.zst symbol archives in mach artifact. r=nalexander,firefox-build-system-reviewers,rstewart
Differential Revision: https://phabricator.services.mozilla.com/D85249
2020-07-30 15:53:26 +00:00
Mike Hommey ddea6bd3ce Bug 1654994 - Support creating .tar.zst archives instead of .zip for symbol archives. r=firefox-build-system-reviewers,rstewart,mhentges
Differential Revision: https://phabricator.services.mozilla.com/D85248
2020-07-30 15:52:51 +00:00
Mitchell Hentges d3dfb78af4 Bug 1656303: Only print $A_SPECIFIC_TARGET warning if one of the targets is not in the allowlist r=firefox-build-system-reviewers,rstewart
The "what" value contains a list of build targets, the warning should be printed if any one of the targets is unexpected.

Differential Revision: https://phabricator.services.mozilla.com/D85462
2020-07-30 19:38:42 +00:00
Mike Hommey f70f000a4f Bug 1614626 - Move xpcshell selftests to before running actual xpcshell tests. r=gbrown
This allows to test them in the exact same environment as the tests are
going to run, which turns out to have revealed a few issues that would
only appear once xpcshell tests fail, impeding on debugging those
failures.

Differential Revision: https://phabricator.services.mozilla.com/D84781
2020-07-30 20:27:14 +00:00
Ricky Stewart 63f010c613 Bug 1646406 - Distinguish between user errors and actual reportable exceptions in `mach` r=mhentges,froydnj
Add a new `UserError` class which when thrown doesn't get reported to Sentry.

Differential Revision: https://phabricator.services.mozilla.com/D85026
2020-07-29 21:34:17 +00:00
Tarek Ziadé cb00bce720 Bug 1655956 - removed unused perf tuning code r=sparky
Removes unused API and add missing coverage

Differential Revision: https://phabricator.services.mozilla.com/D85285
2020-07-29 17:35:35 +00:00
Csoregi Natalia b0e102995f Backed out 2 changesets (bug 1646406, bug 1655529) for geckoview failures. CLOSED TREE
Backed out changeset 0630f4154c62 (bug 1655529)
Backed out changeset 18d42084d135 (bug 1646406)
2020-07-30 00:28:59 +03:00
Ricky Stewart ffd5064859 Bug 1646406 - Distinguish between user errors and actual reportable exceptions in `mach` r=mhentges,froydnj
Add a new `UserError` class which when thrown doesn't get reported to Sentry.

Differential Revision: https://phabricator.services.mozilla.com/D85026
2020-07-29 19:34:24 +00:00
Dorel Luca c24650c12a Backed out changeset 929e1acc42af (bug 1614626) for XPCshell failures in extensions/test/xpcshell/test_ext_dns.js. CLOSED TREE 2020-07-29 07:37:28 +03:00
Mike Hommey efebeaa2c6 Bug 1614626 - Move xpcshell selftests to before running actual xpcshell tests. r=gbrown
This allows to test them in the exact same environment as the tests are
going to run, which turns out to have revealed a few issues that would
only appear once xpcshell tests fail, impeding on debugging those
failures.

Differential Revision: https://phabricator.services.mozilla.com/D84781
2020-07-29 00:23:03 +00:00
Andrew Halberstadt cc0b1f8e5f Bug 985141 - [mozperftest] Fix missed instances of _activate_virtualenv in mozperftest framework, r=perftest-reviewers,sparky
Differential Revision: https://phabricator.services.mozilla.com/D85203
2020-07-28 20:54:58 +00:00
Andrew Halberstadt bef34ebc4b Bug 1408051 - Backed out changeset 7864e0d0295f
This change had a few regressions that will be easier to fix without the pressure of time.

Differential Revision: https://phabricator.services.mozilla.com/D85193
2020-07-28 19:04:53 +00:00
Ricky Stewart da3fe81bea Bug 1655781- Allow specifying that certain packages should only be included in a `virtualenv` when parsing `virtualenv_packages.txt` r=chutten,nalexander
This solves the same problem we attempted to solve in bug 1654663. That was a low-cost, sensible solution when there was only one in-build reference to `glean_parser`, but with project FOG we're about to drastically increase the in-build reliance on the library, so the ad-hoc `sys.path` manipulation is an increasingly insensible solution. Here we address this in a first-class way by specifying that `glean_parser` should be imported in `virtualenv`s, but NOT by top-level `mach` commands that run outside of an in-`objdir` `virtualenv`.

Differential Revision: https://phabricator.services.mozilla.com/D85182
2020-07-28 18:29:55 +00:00
Andrew Halberstadt d0fbcca0b6 Bug 985141 - [mozbuild] Remove leading underscore from MozbuildObject._activate_virtualenv, r=firefox-build-system-reviewers,perftest-reviewers,andi,AlexandruIonescu,rstewart
This function is used all across the tree and should be considered a public API.

Differential Revision: https://phabricator.services.mozilla.com/D85045
2020-07-28 16:06:10 +00:00
Philip Chimento c936b44447 Bug 1653560 - Fix distutils.spawn import. r=andi
Differential Revision: https://phabricator.services.mozilla.com/D84023
2020-07-28 14:54:32 +00:00
Andi-Bogdan Postelnicu f2a6f56b99 Bug 1655701 - for Coverity static-analysis do not relay on ensure exit code. r=marco
Differential Revision: https://phabricator.services.mozilla.com/D85133
2020-07-28 12:52:17 +00:00
Mike Hommey 2a4eb30ed4 Bug 1654465 - Set -Cembed-bitcode=yes instead of CARGO_PROFILE_RELEASE_LTO. r=firefox-build-system-reviewers,rstewart
It turns out setting CARGO_PROFILE_RELEASE_LTO has unwanted side
effects.

First it's not actually strictly equivalent to using `cargo rustc --
-Clto`. For instance, it apparently also enables cross-language LTO in
newer versions of cargo.

Second, it changes the rust computed hash for all the dependencies of
the crate being built with the variable set, which makes them diverge
from when the same dependencies are built through another crate in the
tree that is not LTOed. This effectively makes us build a _lot_ of
crates twice, many of which are not cacheable.

Since the original problem is that cargo >= 1.45 passes extra flags (`-C
embed-bitcode=no`) to rustc that are incompatible with `-Clto`, and while
it knows to adjust based on the `lto` setting in the build profile
(which CARGO_PROFILE_RELEASE_LTO overrides the default of), cargo
ignores flags passed via `cargo rustc -- ...` when making those
adjustments.

So, we need to override with `-C embed-bitcode=yes` on our own at the
same time we pass `-Clto`. But doing that through `cargo rustc -- ...`
is not enough because all the dependencies of the crate built with
`-Clto` need to be built with `-C embed-bitcode=yes`. So we need to
override with `RUSTFLAGS`, which will affect all the dependencies.
But we also need to do this consistently across all crates, not only the
dependencies of crates built with `-Clto`, otherwise we'd still end up
building crates twice (once with and once without the override).

Unfortunately, the `-C embed-bitcode=*` flag is also not supported in
versions older than 1.45, so we have to avoid adding it on older
versions.

We unfortunately support a large range of versions of rustc (albeit only
for tools/crashreporter), but we actually need to upgrade the smaller
supported version because rustc < 1.38 doesn't support our top-level
Cargo.lock. This makes the version check slightly less awful.

Differential Revision: https://phabricator.services.mozilla.com/D84652
2020-07-27 21:23:57 +00:00
Ricky Stewart 3e84200ad6 Bug 1655340 - Install `self.packages` in `mozboot/debian.py` r=froydnj
This got trampled in an overzealous code deletion in bug 1647814.

Differential Revision: https://phabricator.services.mozilla.com/D85004
2020-07-27 15:57:16 +00:00
Mike Hommey f8b53d8e93 Bug 1654961 - Re-enable some python tests on py2 on mac. r=dmajor
Bug 1648506 disabled tests that failed on py2 on mac. Some now pass
properly, presumably from more recent changes to
build/moz.configure/init.configure.

Differential Revision: https://phabricator.services.mozilla.com/D84776
2020-07-24 05:40:41 +00:00
Coroiu Cristina 828f2319c0 Backed out changeset 2439a32e1a06 (bug 1654961) for mbu failures 2020-07-24 08:19:33 +03:00
Mike Hommey 29e887fea7 Bug 1654961 - Re-enable some python tests on py2 on mac. r=dmajor
Bug 1648506 disabled tests that failed on py2 on mac. They now pass
properly, presumably from more recent changes to
build/moz.configure/init.configure.

Differential Revision: https://phabricator.services.mozilla.com/D84776
2020-07-24 04:19:51 +00:00
Tom Prince 97a298b135 Bug 1652503: [mozlint] Use `attrs` for `mozlint.result.Issue`; r=ahal
This gives sorting on `Issue` for free, which makes it easier to write tests
for linters.

Differential Revision: https://phabricator.services.mozilla.com/D84643
2020-07-23 14:55:45 +00:00
Andrew Creskey c6c9bc4f6f Bug 1648857 - Add applink perftest for the reference browser r=tarek,sparky
The Reference Browser represents our tech stack up to and including the Android-Components layer.
It would be useful to have an applink startup test for this browser to detect improvements and regressions.

Differential Revision: https://phabricator.services.mozilla.com/D81407
2020-07-23 14:11:38 +00:00
Emilio Cobos Álvarez a8f027f4ff Bug 1654795 - Fix mach clang-format python3 error. r=andi
Differential Revision: https://phabricator.services.mozilla.com/D84691
2020-07-23 14:06:14 +00:00
Francesco Lodolo (:flod) 3be1f9c7e0 Bug 1654177 - Remove Fluent migration recipes for Firefox 78, r=Pike
Differential Revision: https://phabricator.services.mozilla.com/D84294
2020-07-23 08:03:48 +00:00
Mitchell Hentges 81ec118ee7 Bug 1654607: Upgrades glean in bootstrap r=firefox-build-system-reviewers,rstewart
A newer version of Glean is needed to take advantage of the recent "optional version" change.

Pins glean to 31.5.0 to avoid breaking changes, such as to the YAML schema.

Differential Revision: https://phabricator.services.mozilla.com/D84566
2020-07-22 19:48:23 +00:00
Nathan Froyd a8ddf893b6 Bug 1654633 - install new dump_syms during bootstrap on OS X; r=nalexander,mhentges
This change brings Mac into parity with our other development platforms.

Differential Revision: https://phabricator.services.mozilla.com/D84580
2020-07-22 19:11:45 +00:00
Hamzah 848d6992f5 Bug 1408051 - Remove mozharness' copy of virtualenv and use the one under /third_party/python instead r=ahal,rail
Differential Revision: https://phabricator.services.mozilla.com/D83181
2020-07-17 19:21:18 +00:00
Hamzah 570e9a50fc Bug 1606475 - Move mach clang-format to Python 3 r=andi
Manually tested various combination of commands for `./mach clang-format`. It works fine on both py2 as well as py3

Differential Revision: https://phabricator.services.mozilla.com/D79330
2020-07-20 12:12:28 +00:00
Mike Hommey 268a226874 Bug 1654182 - Only look at the dynamic symbols table for the version checks. r=froydnj
The main reason we look at the complete symbols table is that before bug
1541792, we needed to look at that table for _NSModule symbols.

In bug 1516228, we also made everything llvm-objdump to limit the
differences cross-platform, but that's not necessary anymore per the
previous change.

llvm-objdump doesn't support getting only the dynamic symbols table, so
we go back to what we were using before bug 1516228, namely readelf,
while preserving a code path to use the complete symbols table for the
networking test on libgkrust.a, which doesn't have a dynamic symbols
table.

With this change, check_binary goes from 45s to 0.2s on my machine.

Differential Revision: https://phabricator.services.mozilla.com/D84305
2020-07-22 04:09:39 +00:00
Mike Hommey 7a304e4f58 Bug 1654182 - Remove support for iterating symbols on non-Linux platforms. r=froydnj
The way check_binary currently works is the result of many years of
changes, but some of the more recent ones actually make some of the
earliest ones unnecessary.

For instance, bug 1541792 removed the need to use iter_symbols on
non-Linux platforms. So we remove support for non-Linux (non-ELF,
really) platforms in iter_symbols (and rename the function to reflect
that).

Differential Revision: https://phabricator.services.mozilla.com/D84304
2020-07-22 04:09:21 +00:00
Mike Hommey 42410ac80f Bug 1654182 - Reduce the amount of work done for binary compatibility check. r=froydnj
Iterating symbols is actually expensive because of all the line
splitting. So iterating over all symbols 4 times is actually a huge
waste of time.

Instead, iterate over them once, doing all the version checks at once
for each of them.

This brings down the time spent in check_binary from 2:25 to 0:45 on my
machine.

Differential Revision: https://phabricator.services.mozilla.com/D84303
2020-07-22 00:27:08 +00:00
Dorel Luca 3f7ad38666 Backed out 4 changesets (bug 1654182) for Spidermonkey failure and Android build bustage. CLOSED TREE
Backed out changeset 8ecbb1da91cf (bug 1654182)
Backed out changeset 205329cc5f30 (bug 1654182)
Backed out changeset 13ab8f6d24bf (bug 1654182)
Backed out changeset 6100f0d3c0ad (bug 1654182)
2020-07-22 03:04:57 +03:00
Cosmin Sabou 25f4621719 Bug 1654182 - Fix spidermonkey bustages. r=glandium 2020-07-22 01:54:38 +03:00
Ricky Stewart ffb40eb3c2 Bug 1636797 - Tweak `git clean` call in `mach clobber python` r=mhentges,jgilbert,froydnj
Differential Revision: https://phabricator.services.mozilla.com/D84440
2020-07-21 22:27:21 +00:00
Mike Hommey 06ab4ddb50 Bug 1654182 - Only look at the dynamic symbols table for the version checks. r=froydnj
The main reason we look at the complete symbols table is that before bug
1541792, we needed to look at that table for _NSModule symbols.

In bug 1516228, we also made everything llvm-objdump to limit the
differences cross-platform, but that's not necessary anymore per the
previous change.

llvm-objdump doesn't support getting only the dynamic symbols table, so
we go back to what we were using before bug 1516228, namely readelf,
while preserving a code path to use the complete symbols table for the
networking test on libgkrust.a, which doesn't have a dynamic symbols
table.

With this change, check_binary goes from 45s to 0.2s on my machine.

Differential Revision: https://phabricator.services.mozilla.com/D84305
2020-07-21 13:28:51 +00:00
Mike Hommey a59a02755b Bug 1654182 - Remove support for iterating symbols on non-Linux platforms. r=froydnj
The way check_binary currently works is the result of many years of
changes, but some of the more recent ones actually make some of the
earliest ones unnecessary.

For instance, bug 1541792 removed the need to use iter_symbols on
non-Linux platforms. So we remove support for non-Linux (non-ELF,
really) platforms in iter_symbols (and rename the function to reflect
that).

Differential Revision: https://phabricator.services.mozilla.com/D84304
2020-07-21 21:00:50 +00:00
Mike Hommey 9af7ceef05 Bug 1654182 - Reduce the amount of work done for binary compatibility check. r=froydnj
Iterating symbols is actually expensive because of all the line
splitting. So iterating over all symbols 4 times is actually a huge
waste of time.

Instead, iterate over them once, doing all the version checks at once
for each of them.

This brings down the time spent in check_binary from 2:25 to 0:45 on my
machine.

Differential Revision: https://phabricator.services.mozilla.com/D84303
2020-07-21 22:15:51 +00:00
Ricky Stewart ba024b583c Bug 1651820 - Fix broken xpcshell artifact builds by pointing directly to the `http3server` path r=gbrown
Also make sure we distribute the `http3server` binary in the common test archive, and download it for artifact builds.

Differential Revision: https://phabricator.services.mozilla.com/D84421
2020-07-21 20:47:27 +00:00
Yue Zhang 04a60db9fe Bug 1635930 - Fix notebookupload.py layer. r=sparky
Differential Revision: https://phabricator.services.mozilla.com/D83726
2020-07-21 18:45:40 +00:00
Gregory Mierzwinski 14830697f7 Bug 1652979 - Use org.mozilla.fenix package name in mozperftest. r=tarek
Differential Revision: https://phabricator.services.mozilla.com/D84375
2020-07-21 17:07:00 +00:00
Tarek Ziadé cb88926dd6 Bug 1651624 - Add a macOS layer r=sparky
Adds a macOS layer that provides the ability to
mount DMGs on the fly. For instance Firefox's distribution.

Differential Revision: https://phabricator.services.mozilla.com/D84162
2020-07-21 12:51:58 +00:00
Ting-Yu Lin fdc00547f6 Bug 1637130 - Rename visual overflow to ink overflow. r=dbaron
This patch is generated by:

```
# Rename the nsOverflowType enum.
rg -l "eVisualOverflow" layout/ gfx/ | xargs sed -i "s/eVisualOverflow/eInkOverflow/g"

# Rename and drop the "Get" prefix from various functions.
rg -l "GetVisualOverflowRect" layout/ gfx/ | xargs sed -i "s/GetVisualOverflowRect/InkOverflowRect/g"
rg -l "GetPreEffectsVisualOverflowRect" layout/ gfx/ | xargs sed -i "s/GetPreEffectsVisualOverflowRect/PreEffectsInkOverflowRect/g"
rg -l "GetVisualOverflowFromDeltas" layout/ gfx/ | xargs sed -i "s/GetVisualOverflowFromDeltas/InkOverflowFromDeltas/g"
rg -l "GetScrollableOverflowRect" layout/ gfx/ | xargs sed -i "s/GetScrollableOverflowRect/ScrollableOverflowRect/g"

# Rename, drop the "Get" prefix, and change the suffix "Area" to "Rect"
# (because they return nsRect) for the two methods in nsLineBox.
rg -l "GetVisualOverflowArea" layout/ gfx/ | xargs sed -i "s/GetVisualOverflowArea/InkOverflowRect/g"
rg -l "GetScrollableOverflowArea" layout/ gfx/ | xargs sed -i "s/GetScrollableOverflowArea/ScrollableOverflowRect/g"

# Rename rest of the functions and variables.
rg -l "VisualOverflow" layout/ gfx/ | xargs sed -i "s/VisualOverflow/InkOverflow/g"
rg -l "visual overflow" layout/ gfx/ | xargs sed -i "s/visual overflow/ink overflow/g"
rg -l "visualOverflow" layout/ gfx/ | xargs sed -i "s/visualOverflow/inkOverflow/g"
rg -l "visOverflow" layout/ gfx/ | xargs sed -i "s/visOverflow/inkOverflow/g"
rg -l "vis-overflow" layout/ gfx/ python/ | xargs sed -i "s/vis-overflow/ink-overflow/g"

./mach clang-format
```

Differential Revision: https://phabricator.services.mozilla.com/D84231
2020-07-20 20:17:36 +00:00
Ricky Stewart 31cd1c2dbc Bug 1647814 - Verify that Python is installed and up-to-date in `bootstrap`. r=mhentges,froydnj
Check whether Python 3 is up-to-date and bail out with specific installation guidance if not. Don't install or upgrade Python 2/3. Also check whether Python 2 is up-to-date, and while we don't block `bootstrap` on it (because Python 2 isn't required to build), also supplement with specific installation guidance for people who need it.

Differential Revision: https://phabricator.services.mozilla.com/D82739
2020-07-20 18:15:06 +00:00
Bob Clary 6b2b24ee30 Bug 1486004 - Part 6 - Update mach perftest to support unrooted android devices using mozdevice 4.0.0, ? r=tarek
Depends on D82748

Differential Revision: https://phabricator.services.mozilla.com/D82749
2020-07-17 20:48:35 +00:00
Bob Clary 15be5895b3 Bug 1486004 - Part 3 - change help prompt for remote test root to show /data/local/tmp/test_root, r=gbrown
Depends on D82745

Differential Revision: https://phabricator.services.mozilla.com/D82746
2020-07-17 20:48:42 +00:00
Johan Lorenzo b651a3366d Bug 1631839 - Update mobile routes to use new namespace r=tarek,sparky,acreskey
Differential Revision: https://phabricator.services.mozilla.com/D81038
2020-07-17 16:00:57 +00:00
Tom Prince 7524e40b38 Bug 1653662: [mozrelease] Update mozrelease subcommands to python 3; r=aki
Differential Revision: https://phabricator.services.mozilla.com/D84042
2020-07-18 01:36:36 +00:00
Tarek Ziadé df3af8361c Bug 1651300 - support for xpcshell tests r=sparky
Implements xpcshell support, through the new flavor

Differential Revision: https://phabricator.services.mozilla.com/D82727
2020-07-17 22:12:26 +00:00
Tarek Ziadé 883db95e9d Bug 1628278 - added conditioned profile support r=acreskey
Adds conditioned profiles and make the Profile
layer a system layer instead of a test layer

Differential Revision: https://phabricator.services.mozilla.com/D83612
2020-07-16 13:30:11 +00:00
Narcis Beleuzu b1faa2872a Backed out 17 changesets (bug 1486004) for wrench bustage on android.py . CLOSED TREE
Backed out changeset cae730f105d7 (bug 1486004)
Backed out changeset e0094d50a315 (bug 1486004)
Backed out changeset 424425ed98c6 (bug 1486004)
Backed out changeset 551232749abe (bug 1486004)
Backed out changeset 505747095def (bug 1486004)
Backed out changeset 5b1c62c4a4d5 (bug 1486004)
Backed out changeset 0a1c8d50ee69 (bug 1486004)
Backed out changeset 6ef81c676859 (bug 1486004)
Backed out changeset 234e55153fb6 (bug 1486004)
Backed out changeset 2f33b309d746 (bug 1486004)
Backed out changeset 48947692c176 (bug 1486004)
Backed out changeset 473bac8eedd9 (bug 1486004)
Backed out changeset af6afed0a04a (bug 1486004)
Backed out changeset a0e2f91eb4d4 (bug 1486004)
Backed out changeset 138c987440c7 (bug 1486004)
Backed out changeset 6d2e305f4722 (bug 1486004)
Backed out changeset 1181ff03bb05 (bug 1486004)
2020-07-17 23:43:57 +03:00
Bob Clary d667fb0fa1 Bug 1486004 - Part 6 - Update mach perftest to support unrooted android devices using mozdevice 4.0.0, ? r=tarek
Depends on D82748

Differential Revision: https://phabricator.services.mozilla.com/D82749
2020-07-16 21:32:53 +00:00
Bob Clary fc8e9dbc87 Bug 1486004 - Part 3 - change help prompt for remote test root to show /data/local/tmp/test_root, r=gbrown
Depends on D82745

Differential Revision: https://phabricator.services.mozilla.com/D82746
2020-07-16 21:32:52 +00:00
Mitchell Hentges 136fa3e403 Bug 1653135: Sentry integration should handle incomplete settings r=firefox-build-system-reviewers,rstewart
In tests, the settings object doesn't have as many sections. When a section isn't
available, it raises an error.
The Sentry integration can interpret this error as telemetry being disabled.

Differential Revision: https://phabricator.services.mozilla.com/D83717
2020-07-16 19:27:15 +00:00
Ricky Stewart f34f1b6153 Bug 1648555 - `mach bootstrap` installs `libdrm-dev` package on Ubuntu/Debian r=dmajor
Also delete a bunch of dead code that was apparently meant to be useful for subclassing, but there's no subclasses of `DebianBootstrapper` in tree.

Differential Revision: https://phabricator.services.mozilla.com/D82073
2020-07-16 13:49:59 +00:00
Narcis Beleuzu 1eb115035d Backed out 20 changesets (bug 1486004, bug 1653094) for wrench bustages. CLOSED TREE
Backed out changeset 120c64ebdea3 (bug 1486004)
Backed out changeset b10f105b1c36 (bug 1486004)
Backed out changeset 7f186e6bcccc (bug 1486004)
Backed out changeset cdda7b349cd8 (bug 1486004)
Backed out changeset 650c47e81402 (bug 1486004)
Backed out changeset 2bcca7cf3f1b (bug 1486004)
Backed out changeset e1df276f8a8d (bug 1486004)
Backed out changeset f6f5e7f2bd0a (bug 1486004)
Backed out changeset 1c56cb03ce53 (bug 1486004)
Backed out changeset 582cd190f174 (bug 1486004)
Backed out changeset 7c162e1b3a79 (bug 1486004)
Backed out changeset d1088e178216 (bug 1486004)
Backed out changeset 40763f4afd26 (bug 1486004)
Backed out changeset f20eb5b5a670 (bug 1486004)
Backed out changeset 6349ee0bd63f (bug 1486004)
Backed out changeset 7c23729053f6 (bug 1486004)
Backed out changeset 49586014bd29 (bug 1486004)
Backed out changeset 07971e82ae85 (bug 1653094)
Backed out changeset 47e7295e9c19 (bug 1653094)
Backed out changeset 57a25bbeec37 (bug 1653094)
2020-07-16 17:58:09 +03:00
Bob Clary 8f3e6b9e4c Bug 1486004 - Part 6 - Update mach perftest to support unrooted android devices using mozdevice 4.0.0, ? r=tarek
Depends on D82748

Differential Revision: https://phabricator.services.mozilla.com/D82749
2020-07-15 20:58:20 +00:00
Bob Clary aaa88f81bd Bug 1486004 - Part 3 - change help prompt for remote test root to show /data/local/tmp/test_root, r=gbrown
Depends on D82745

Differential Revision: https://phabricator.services.mozilla.com/D82746
2020-07-15 20:57:32 +00:00
Mihai Alexandru Michis a1d49fb98d Backed out 17 changesets (bug 1486004) for causing bustages in AccessCheck.h
CLOSED TREE

Backed out changeset 86a0855e1e45 (bug 1486004)
Backed out changeset 4fc98067edc5 (bug 1486004)
Backed out changeset 8ab3f1597fde (bug 1486004)
Backed out changeset b487e98320fb (bug 1486004)
Backed out changeset 4cba8e6bc515 (bug 1486004)
Backed out changeset d0db5eef2343 (bug 1486004)
Backed out changeset 50285a87de6b (bug 1486004)
Backed out changeset 9730c0303810 (bug 1486004)
Backed out changeset 140a9d2362f3 (bug 1486004)
Backed out changeset 9d1ce06f6039 (bug 1486004)
Backed out changeset 88a5a4ab39fa (bug 1486004)
Backed out changeset eb186c92232d (bug 1486004)
Backed out changeset 677f2129750e (bug 1486004)
Backed out changeset 81427a5f2103 (bug 1486004)
Backed out changeset 38d7def3d425 (bug 1486004)
Backed out changeset a9e73221c471 (bug 1486004)
Backed out changeset ebe8c2d5432e (bug 1486004)
2020-07-16 15:29:29 +03:00
Bob Clary bcf4eeb0d5 Bug 1486004 - Part 6 - Update mach perftest to support unrooted android devices using mozdevice 4.0.0, ? r=tarek
Depends on D82748

Differential Revision: https://phabricator.services.mozilla.com/D82749
2020-07-15 20:58:20 +00:00
Bob Clary ca55ed23c7 Bug 1486004 - Part 3 - change help prompt for remote test root to show /data/local/tmp/test_root, r=gbrown
Depends on D82745

Differential Revision: https://phabricator.services.mozilla.com/D82746
2020-07-15 20:57:32 +00:00
Mitchell Hentges 115500932d Bug 1651424: Install glean in bootstrap r=firefox-build-system-reviewers,rstewart
Installs glean to the currently-active python installation during bootstrap, whether
that be a virtualenv or the system instance of python.

Differential Revision: https://phabricator.services.mozilla.com/D83012
2020-07-15 22:50:11 +00:00
Mitchell Hentges 301f146dc4 Bug 1652615: Share is_telemetry_enabled() logic between Sentry and telemetry r=firefox-build-system-reviewers,rstewart
Note that this moves settings-loading earlier in the bootstrap procedure,
from "_run()" to "run()".

Differential Revision: https://phabricator.services.mozilla.com/D83425
2020-07-15 23:08:17 +00:00
Myeongjun Go 32cbdb3b33 Bug 1649497 - [perfdocs] Port mozperftest into Performance Testing r=sparky,perftest-reviewers
Differential Revision: https://phabricator.services.mozilla.com/D82207
2020-07-15 17:13:30 +00:00
Mihai Alexandru Michis 961794719d Merge mozilla-central to autoland a=merge on a CLOSED TREE 2020-07-15 04:01:47 +03:00
Tom Prince e0998f9228 Bug 1652786: [taskgraph] Fix parameter handling `load_tasks_for_kind; r=Callek a=Callek
DONTBUILD

Differential Revision: https://phabricator.services.mozilla.com/D83566
2020-07-14 21:09:34 +00:00
Ricky Stewart 9390958b5b Bug 1652736 - Remove import of `mozversioncontrol` which breaks standalone `bootstrap.py` r=nalexander
Differential Revision: https://phabricator.services.mozilla.com/D83557
2020-07-14 20:04:51 +00:00
Csoregi Natalia 2d34e224ae Backed out changeset 448679382b06 (bug 1649497) for linting failures on framework_gatherers.py. CLOSED TREE 2020-07-14 20:20:48 +03:00
Myeongjun Go 62777a18b5 Bug 1649497 - [perfdocs] Port mozperftest into Performance Testing r=sparky,perftest-reviewers
Differential Revision: https://phabricator.services.mozilla.com/D82207
2020-07-14 16:55:56 +00:00
Tom Prince 10f35da9c8 Bug 1646834: [scriptworker-canary] Split linux signing tasks to handle partials; r=rail
Since partials have started verifying signatures, the partial task has been failing in
`mach try scriptworker`. Since we are not concerned with the partial task itself,
split the tasks into two groups, so that it does not need to run.

Differential Revision: https://phabricator.services.mozilla.com/D83370
2020-07-13 18:46:51 +00:00
Tarek Ziadé 7dfca9db55 Bug 1644778 - add a mozperftest to measure browser.xhtml DOM size, r=tarek,dhouse,sparky
Differential Revision: https://phabricator.services.mozilla.com/D79152
2020-07-13 15:44:38 +00:00
Nathan Froyd 95cc11b5d8 Bug 1652087 - add dump_syms to bootstrap-installable things on Linux; r=nalexander
If we're going to ditch the old breakpad dump-syms from the build, people
are going to need have this locally if they ever want to build packages,
etc.

Differential Revision: https://phabricator.services.mozilla.com/D83150
2020-07-10 22:10:26 +00:00
Bob Clary 13b8cbabeb Bug 1652339 - set mozdevice version requirements to exclude version 4.0.0 on trunk, beta r=marionette-reviewers,perftest-reviewers,whimboo,sparky
Differential Revision: https://phabricator.services.mozilla.com/D83278
2020-07-13 11:36:39 +00:00
Mike Hommey 95d2810ac3 Bug 1651287 - Move .inc files to the pre-compile pseudo-tier. r=nalexander
All of them are only used in one source file, in the same directory
where they are generated.

Differential Revision: https://phabricator.services.mozilla.com/D83037
2020-07-10 16:14:34 +00:00
Mike Hommey ee3002aa4e Bug 1651287 - Hook the pre-compile pseudo-tier in the compile graph. r=nalexander
This makes dir/target, dir/target-objects, etc. depend on
dir/pre-compile, which ensures all items in the pre-compile pseudo-tier
are built before entering in the directory they are defined in, during
the compile tier. This removes the need for EXTRA_DEPS hacks.

Differential Revision: https://phabricator.services.mozilla.com/D83036
2020-07-10 21:03:41 +00:00