Граф коммитов

7163 Коммитов

Автор SHA1 Сообщение Дата
Tom Ritter d28e974e6a Bug 1712817: Add tests for mozbuild file editing r=jewilde
Depends on D120481

Differential Revision: https://phabricator.services.mozilla.com/D121379
2021-08-02 18:25:36 +00:00
Tom Ritter 84efe0b813 Bug 1712817: Support adding header files to EXPORTS in mozbuild rewriting r=jewilde
Differential Revision: https://phabricator.services.mozilla.com/D120481
2021-08-02 18:25:35 +00:00
Tom Ritter 292fafda23 Bug 1721247: Lift some repeated code into a function r=jewilde
Differential Revision: https://phabricator.services.mozilla.com/D120480
2021-08-02 18:25:35 +00:00
Tom Ritter a8d5c149f1 Bug 1721247: Ensure that only failure to add/remove exceptions are swallowed r=jewilde
Differential Revision: https://phabricator.services.mozilla.com/D120479
2021-08-02 18:25:34 +00:00
Tom Ritter c18c315a61 Bug 1721247: Improve the mozbuild rewriting logic for different path types r=jewilde
libdav1d only has relative paths. Like ../../third_party/foo/bar.c
libjpeg-xl has absolute paths like /third_party/bar/foo.c

This handles the second case by not doing our relative path
correction logic if we don't need to.

It will find all of the types of ways a file is included in a
moz.build file* and iterate over them, trying the slightly
different logic.  This is okay, because the failure scenario
for trying an incorrect mode is just that we don't get a match.

By trying all the modes we can increase our chances of finding a
match.

* For SOURCES/UNIFIED_SOURCES there aren't actually any situations
where we have mixed-modes in a moz.build file. But a future patch
will add support for editing EXPORTS and they do have mixed-modes.

Differential Revision: https://phabricator.services.mozilla.com/D120478
2021-08-02 18:25:34 +00:00
Tom Ritter 054304451d Bug 1721247: Improve/fix the local testing logic for mach vendor mozbuild rewriting r=jewilde
Differential Revision: https://phabricator.services.mozilla.com/D120477
2021-08-02 18:25:34 +00:00
Nazım Can Altınova 424eaf980b Bug 1652560 - Add bindgen build step to gecko-profiler API crate r=emilio
Bindgen will be needed for gecko_profiler::is_active gecko_profiler_label API
in this patch, as well as the marker API that will land after this.

Differential Revision: https://phabricator.services.mozilla.com/D120789
2021-07-30 21:49:20 +00:00
Alex Lopez a2997cb1c6 Bug 1696251 - Fix error in static analysis mach_commands.py. r=mhentges
When transitioning from using `self` to using `command_context` in mach
commands, this specific mistake went undetected (no bustage forced a backout,
 either). This commit fixes it.

Differential Revision: https://phabricator.services.mozilla.com/D120804
2021-07-30 20:12:39 +00:00
Kagami Sascha Rosylight ca8aae8bc7 Bug 1671221 - Use posix path for static analysis r=andi
Differential Revision: https://phabricator.services.mozilla.com/D121291
2021-07-30 14:08:41 +00:00
Noemi Erli 857763331c Backed out 7 changesets (bug 1720215, bug 1717645, bug 1717051) for causing build bustage and coflicting backout
Backed out changeset d3f2270f5d80 (bug 1717051)
Backed out changeset 06ae6038c823 (bug 1717051)
Backed out changeset 732724b94715 (bug 1717051)
Backed out changeset 51278cadd18c (bug 1720215)
Backed out changeset 766ea0a1adf1 (bug 1717645)
Backed out changeset ea6ab9ecdc1b (bug 1717645)
Backed out changeset 61b5c99a7ad2 (bug 1717645)
2021-07-30 04:00:31 +03:00
Mike Hommey e87d30886d Bug 1722439 - Bootstrap wasi-sysroot via configure. r=firefox-build-system-reviewers,andi
Differential Revision: https://phabricator.services.mozilla.com/D120924
2021-07-29 23:49:33 +00:00
Mitchell Hentges 7ec0e9aff9 Bug 1717051: Move global mach dependencies to requirements definition r=ahal
After removing `optional` in Bug 1712804, we need to add a variant back
here because there's fallible dependencies. However, I've tweaked the
re-introduction of the feature to require a specific repercussion
message as well. This seemed like a decent tradeoff - the developer
becomes aware that the failure is bad, it has repercussions, but it's
not a blocking issue. Additionally, since we're printing pip's output,
the developer will be able to see the underlying error causing the
warning.

I also added comment functionality to requirements definitions to allow
adjacent documentation of why some requirements are fallible. (Related:
I'm looking forward to `mach_bootstrap` not needing to parse
requirements definitions. Almost there!)

Note that we'll temporarily lose the "pinned" nature of the
three moved dependencies until dependency locking is implemented
for Mach requirements definitions. Also note that the pinned
`zstandard_requirements.txt` can't be removed like the other
files because it still has a dangling usage.

Finally, in preparation for review: I didn't make
`PypiOptionalSpecifier` extend `PypiSpecifier` because I figured that
the benefit of flexibility (easier to allow implementations to diverge
without needing to untangle an inheritance relationship) was larger than
the cost of needing to add properties to both specifiers.
If we wanted re-use, I'd probably have `PypiOptionalSpecifier` _contain_
 a `PypiSpecifier`, but then you have to reach deeper into the object to
 get data, so *shrug*.

Differential Revision: https://phabricator.services.mozilla.com/D119835
2021-07-29 23:41:13 +00:00
Mitchell Hentges 606b96966b Bug 1717051: Only use stdlib for installing pip packages r=ahal
`_install_pip_package()` may be run from `populate()`, which
is invoked from a child Python process that doesn't
have in-tree Python modules in its sys.path.

An alternate solution would be to add in-tree modules
to the sys.path, but that seemed more costly than
simply using `tempfile` and `shutil`.

Differential Revision: https://phabricator.services.mozilla.com/D119834
2021-07-29 23:41:12 +00:00
Csoregi Natalia 4bfa1610f2 Backed out 3 changesets (bug 1717051) for causing gecko decision failure. CLOSED TREE
Backed out changeset 1a12fa683af8 (bug 1717051)
Backed out changeset d8de22311420 (bug 1717051)
Backed out changeset b522a5b7a10e (bug 1717051)
2021-07-30 02:15:50 +03:00
Mitchell Hentges b7b8e6f938 Bug 1717051: Move global mach dependencies to requirements definition r=ahal
After removing `optional` in Bug 1712804, we need to add a variant back
here because there's fallible dependencies. However, I've tweaked the
re-introduction of the feature to require a specific repercussion
message as well. This seemed like a decent tradeoff - the developer
becomes aware that the failure is bad, it has repercussions, but it's
not a blocking issue. Additionally, since we're printing pip's output,
the developer will be able to see the underlying error causing the
warning.

I also added comment functionality to requirements definitions to allow
adjacent documentation of why some requirements are fallible. (Related:
I'm looking forward to `mach_bootstrap` not needing to parse
requirements definitions. Almost there!)

Note that we'll temporarily lose the "pinned" nature of the
three moved dependencies until dependency locking is implemented
for Mach requirements definitions. Also note that the pinned
`zstandard_requirements.txt` can't be removed like the other
files because it still has a dangling usage.

Finally, in preparation for review: I didn't make
`PypiOptionalSpecifier` extend `PypiSpecifier` because I figured that
the benefit of flexibility (easier to allow implementations to diverge
without needing to untangle an inheritance relationship) was larger than
the cost of needing to add properties to both specifiers.
If we wanted re-use, I'd probably have `PypiOptionalSpecifier` _contain_
 a `PypiSpecifier`, but then you have to reach deeper into the object to
 get data, so *shrug*.

Differential Revision: https://phabricator.services.mozilla.com/D119835
2021-07-29 22:26:24 +00:00
Mitchell Hentges 25e660e459 Bug 1717051: Only use stdlib for installing pip packages r=ahal
`_install_pip_package()` may be run from `populate()`, which
is invoked from a child Python process that doesn't
have in-tree Python modules in its sys.path.

An alternate solution would be to add in-tree modules
to the sys.path, but that seemed more costly than
simply using `tempfile` and `shutil`.

Differential Revision: https://phabricator.services.mozilla.com/D119834
2021-07-29 22:26:23 +00:00
Mitchell Hentges c5988d8dbb Bug 1720215: Remove MACOSX_DEPLOYMENT_TARGET virtualenv workaround r=firefox-build-system-reviewers,andi
Now that the upstream Python bug has been resolved since Python 3.4 (at
the latest), we can safely remove the environment variable workaround.

Differential Revision: https://phabricator.services.mozilla.com/D119687
2021-07-29 22:26:23 +00:00
Mitchell Hentges ec0e3536bb Bug 1717645: Fix up_to_date for Python 3.6 r=ahal
Python 3.6 doesn't support the `capture_output` option, so use
`stdout=subprocess.PIPE` instead.

Differential Revision: https://phabricator.services.mozilla.com/D120027
2021-07-29 22:26:22 +00:00
Mitchell Hentges fbb9bb362b Bug 1717645: Check virtualenv pths are up-to-date r=ahal
It's possible for a virtualenv to have an incorrect list of directories
to add to the sys.path, such as the following cases:
* Its creation got cancelled halfway through
* The list of pths changed in a new revision
* It got modified by an external tool

By validating the list of provided pths against the list of required
pths, we can ensure that the virtualenv is more dependably up-to-date.

Differential Revision: https://phabricator.services.mozilla.com/D119686
2021-07-29 22:26:22 +00:00
Mitchell Hentges a6f09bc76d Bug 1717645: Resolve nested virtualenv requirements up-front r=ahal
This simplifies consumer logic, since they get the parsed list of pypi
and pth requirements, as well as the list of input files that were
parsed.

One benefit of this simplification is that we no longer
recursively create VirtualenvManagers.

Note that mach_bootstrap cannot (yet) take advantage
of `ParseMachEnvRequirements` because of a dependency cycle:
* `mach_bootstrap` must set up the `sys.path` to import
  `ParseMachEnvRequirements`.
* `mach_bootstrap` would want `ParseMachEnvRequirements` to
  determine which paths to add to the `sys.path`.

Differential Revision: https://phabricator.services.mozilla.com/D119685
2021-07-29 22:26:21 +00:00
Rob Wu c2897f87fd Bug 1722960 - Append "/bin" to JVMHomePath on macOS r=nalexander
Differential Revision: https://phabricator.services.mozilla.com/D121210
2021-07-29 22:00:05 +00:00
Agi Sferro 475c203ebc Bug 1722483 - Remove emulator lock files if present. r=nalexander
Looks like the emulator doesn't run the cleanup step when running in headless
mode, so we need to remove the lock files manually:

84416e78b4/android/android-emu/android/console.cpp (4016)

Differential Revision: https://phabricator.services.mozilla.com/D120999
2021-07-28 17:55:56 +00:00
Agi Sferro 510ffc6824 Bug 1722483 - Rename avd_path -> avd_home_path. r=nalexander
Differential Revision: https://phabricator.services.mozilla.com/D120998
2021-07-28 17:55:56 +00:00
Rob Wu 0d5b1e831a Bug 1722492 - Use JDK on macOS if available r=nalexander
Differential Revision: https://phabricator.services.mozilla.com/D120955
2021-07-27 16:43:04 +00:00
Agi Sferro 0e754e748c Bug 1722347 - Uncomment ensure_*_packages. r=nalexander
This was missed in Bug 1721669, likely a rebase error.

Differential Revision: https://phabricator.services.mozilla.com/D120861
2021-07-26 17:16:03 +00:00
Agi Sferro a193bd2970 Bug 1722347 - Add missing |application| parameter in install_private_packages. r=nalexander
Differential Revision: https://phabricator.services.mozilla.com/D120857
2021-07-26 17:16:03 +00:00
Agi Sferro 1bfe02187f Bug 1718341 - Enable keyboard in Android AVDs. r=nalexander
Just noticed that the emulators don't have the keyboard enabled, which makes it
very annoying to test stuff locally.

Differential Revision: https://phabricator.services.mozilla.com/D120100
2021-07-22 17:38:43 +00:00
Benjamin Beurdouche 4582da2473 Bug 1709817 - Enable NSS documentation in firefox-src-tree. r=beurdouche
Differential Revision: https://phabricator.services.mozilla.com/D119913
2021-07-22 12:10:19 +00:00
Agi Sferro 16cdae67a6 Bug 1721669 - Don't overload ensure_browser_packages. r=nalexander
ensure_browser_packages is used for another purpose since Bug 1718341.

this also fixed a missing overload for ensure_android_artifact_mode_packages
and ensure_js_packages and some missing arguments in install_android_packages.

Differential Revision: https://phabricator.services.mozilla.com/D120511
2021-07-21 20:55:00 +00:00
Butkovits Atila e3cb57c97c Merge mozilla-central to autoland. CLOSED TREE 2021-07-22 00:57:15 +03:00
Francesco Lodolo (:flod) d29380d22c Bug 1721442 - Add support for ESR91 to cross-channel, r=aki DONTBUILD
Differential Revision: https://phabricator.services.mozilla.com/D120370
2021-07-21 17:27:39 +00:00
Sebastian Hengst 2fa4a55f4a Bug 1721538 - comment out call which causes ./mach bootstrap to fail. a=bootstrap-fix
Code got added in bug 1718341 yesterday, fix mentioned by Zibi on #developers.
2021-07-21 16:26:51 +02:00
Sandor Molnar 77256682a1 Backed out 7 changesets (bug 1720215, bug 1717645, bug 1717051) for causing win debug build bustages. CLOSED TREE
Backed out changeset fb9919b26f30 (bug 1717051)
Backed out changeset 605651561f2a (bug 1717051)
Backed out changeset d1333fe69cf4 (bug 1717051)
Backed out changeset b75dcb922521 (bug 1720215)
Backed out changeset e43fb9c6cd0f (bug 1717645)
Backed out changeset eab19a103b56 (bug 1717645)
Backed out changeset 7864e8241d77 (bug 1717645)
2021-07-21 07:47:18 +03:00
Mitchell Hentges 9d2879a48a Bug 1717051: Move global mach dependencies to requirements definition r=ahal
After removing `optional` in Bug 1712804, we need to add a variant back
here because there's fallible dependencies. However, I've tweaked the
re-introduction of the feature to require a specific repercussion
message as well. This seemed like a decent tradeoff - the developer
becomes aware that the failure is bad, it has repercussions, but it's
not a blocking issue. Additionally, since we're printing pip's output,
the developer will be able to see the underlying error causing the
warning.

I also added comment functionality to requirements definitions to allow
adjacent documentation of why some requirements are fallible. (Related:
I'm looking forward to `mach_bootstrap` not needing to parse
requirements definitions. Almost there!)

Note that we'll temporarily lose the "pinned" nature of the
three moved dependencies until dependency locking is implemented
for Mach requirements definitions. Also note that the pinned
`zstandard_requirements.txt` can't be removed like the other
files because it still has a dangling usage.

Finally, in preparation for review: I didn't make
`PypiOptionalSpecifier` extend `PypiSpecifier` because I figured that
the benefit of flexibility (easier to allow implementations to diverge
without needing to untangle an inheritance relationship) was larger than
the cost of needing to add properties to both specifiers.
If we wanted re-use, I'd probably have `PypiOptionalSpecifier` _contain_
 a `PypiSpecifier`, but then you have to reach deeper into the object to
 get data, so *shrug*.

Differential Revision: https://phabricator.services.mozilla.com/D119835
2021-07-20 22:22:53 +00:00
Mitchell Hentges f4c722d4cf Bug 1717051: Only use stdlib for installing pip packages r=ahal
`_install_pip_package()` may be run from `populate()`, which
is invoked from a child Python process that doesn't
have in-tree Python modules in its sys.path.

An alternate solution would be to add in-tree modules
to the sys.path, but that seemed more costly than
simply using `tempfile` and `shutil`.

Differential Revision: https://phabricator.services.mozilla.com/D119834
2021-07-20 22:22:53 +00:00
Mitchell Hentges b34427450f Bug 1720215: Remove MACOSX_DEPLOYMENT_TARGET virtualenv workaround r=firefox-build-system-reviewers,andi
Now that the upstream Python bug has been resolved since Python 3.4 (at
the latest), we can safely remove the environment variable workaround.

Differential Revision: https://phabricator.services.mozilla.com/D119687
2021-07-20 22:22:52 +00:00
Mitchell Hentges c958faf669 Bug 1717645: Fix up_to_date for Python 3.6 r=ahal
Python 3.6 doesn't support the `capture_output` option, so use
`stdout=subprocess.PIPE` instead.

Differential Revision: https://phabricator.services.mozilla.com/D120027
2021-07-20 22:22:52 +00:00
Mitchell Hentges 6ae508148e Bug 1717645: Check virtualenv pths are up-to-date r=ahal
It's possible for a virtualenv to have an incorrect list of directories
to add to the sys.path, such as the following cases:
* Its creation got cancelled halfway through
* The list of pths changed in a new revision
* It got modified by an external tool

By validating the list of provided pths against the list of required
pths, we can ensure that the virtualenv is more dependably up-to-date.

Differential Revision: https://phabricator.services.mozilla.com/D119686
2021-07-20 22:22:51 +00:00
Mitchell Hentges d11e0a94e8 Bug 1717645: Resolve nested virtualenv requirements up-front r=ahal
This simplifies consumer logic, since they get the parsed list of pypi
and pth requirements, as well as the list of input files that were
parsed.

One benefit of this simplification is that we no longer
recursively create VirtualenvManagers.

Note that mach_bootstrap cannot (yet) take advantage
of `ParseMachEnvRequirements` because of a dependency cycle:
* `mach_bootstrap` must set up the `sys.path` to import
  `ParseMachEnvRequirements`.
* `mach_bootstrap` would want `ParseMachEnvRequirements` to
  determine which paths to add to the `sys.path`.

Differential Revision: https://phabricator.services.mozilla.com/D119685
2021-07-20 22:22:51 +00:00
Agi Sferro e8f6769032 Bug 1718341 - Download Android AVD from TL job artifact. r=nalexander
Differential Revision: https://phabricator.services.mozilla.com/D119542
2021-07-20 22:19:16 +00:00
Agi Sferro 83fffa1198 Bug 1718341 - Update Android cmdtools to 4.0. r=nalexander,glandium
The SDK Tools version we're currently using is deprecated and was last released
in 2017. It's time for an upgrade!

Differential Revision: https://phabricator.services.mozilla.com/D119226
2021-07-20 22:19:16 +00:00
Agi Sferro 43e1e251c7 Bug 1718341 - Remove redundant override in MobileAndroidBootstrapper. r=nalexander
Differential Revision: https://phabricator.services.mozilla.com/D119998
2021-07-20 22:19:15 +00:00
Agi Sferro 4391056abd Bug 1718341 - Prewarm AVD before packaging r=nalexander
The very first time an Android AVD starts it runs some one time jobs to
properly set up the AVD.

To avoid running the setup every time we run tests in automation, we can boot
the AVD before packaging it so that the testing jobs can use a "prewarmed" AVD
instead.

Differential Revision: https://phabricator.services.mozilla.com/D119225
2021-07-20 22:19:15 +00:00
Agi Sferro 7517b3ca00 Bug 1718341 - Generate Android AVD in TL job instead of using tooltool r=aklotz,nalexander
Tooltool images are hard to update because we don't provide a script to
generate the image and documentation is often inaccurate.

This patch makes it so we generate the AVD in the android-sdk TL job instead.

Differential Revision: https://phabricator.services.mozilla.com/D119221
2021-07-20 22:19:14 +00:00
Cosmin Sabou 84b8cec8b2 Backed out 7 changesets (bug 1717645, bug 1717051, bug 1720215) for busting the gecko decision task. CLOSED TREE DONTBUILD
Backed out changeset dbdd100c60d0 (bug 1717051)
Backed out changeset 4f32438176a0 (bug 1717051)
Backed out changeset 6597db537fa2 (bug 1717051)
Backed out changeset cc686ee3fb6e (bug 1720215)
Backed out changeset 82dc0771b98b (bug 1717645)
Backed out changeset 16b0fb30aff2 (bug 1717645)
Backed out changeset 370e02f73923 (bug 1717645)
2021-07-21 00:59:41 +03:00
Mitchell Hentges ffaaa6b319 Bug 1717051: Move global mach dependencies to requirements definition r=ahal
After removing `optional` in Bug 1712804, we need to add a variant back
here because there's fallible dependencies. However, I've tweaked the
re-introduction of the feature to require a specific repercussion
message as well. This seemed like a decent tradeoff - the developer
becomes aware that the failure is bad, it has repercussions, but it's
not a blocking issue. Additionally, since we're printing pip's output,
the developer will be able to see the underlying error causing the
warning.

I also added comment functionality to requirements definitions to allow
adjacent documentation of why some requirements are fallible. (Related:
I'm looking forward to `mach_bootstrap` not needing to parse
requirements definitions. Almost there!)

Note that we'll temporarily lose the "pinned" nature of the
three moved dependencies until dependency locking is implemented
for Mach requirements definitions. Also note that the pinned
`zstandard_requirements.txt` can't be removed like the other
files because it still has a dangling usage.

Finally, in preparation for review: I didn't make
`PypiOptionalSpecifier` extend `PypiSpecifier` because I figured that
the benefit of flexibility (easier to allow implementations to diverge
without needing to untangle an inheritance relationship) was larger than
the cost of needing to add properties to both specifiers.
If we wanted re-use, I'd probably have `PypiOptionalSpecifier` _contain_
 a `PypiSpecifier`, but then you have to reach deeper into the object to
 get data, so *shrug*.

Differential Revision: https://phabricator.services.mozilla.com/D119835
2021-07-20 21:42:04 +00:00
Mitchell Hentges 16b857cec1 Bug 1717051: Only use stdlib for installing pip packages r=ahal
`_install_pip_package()` may be run from `populate()`, which
is invoked from a child Python process that doesn't
have in-tree Python modules in its sys.path.

An alternate solution would be to add in-tree modules
to the sys.path, but that seemed more costly than
simply using `tempfile` and `shutil`.

Differential Revision: https://phabricator.services.mozilla.com/D119834
2021-07-20 21:42:03 +00:00
Mitchell Hentges 82436a7373 Bug 1720215: Remove MACOSX_DEPLOYMENT_TARGET virtualenv workaround r=firefox-build-system-reviewers,andi
Now that the upstream Python bug has been resolved since Python 3.4 (at
the latest), we can safely remove the environment variable workaround.

Differential Revision: https://phabricator.services.mozilla.com/D119687
2021-07-20 21:42:03 +00:00
Mitchell Hentges 9b2d50e981 Bug 1717645: Fix up_to_date for Python 3.6 r=ahal
Python 3.6 doesn't support the `capture_output` option, so use
`stdout=subprocess.PIPE` instead.

Differential Revision: https://phabricator.services.mozilla.com/D120027
2021-07-20 21:42:02 +00:00
Mitchell Hentges c12ca0aff6 Bug 1717645: Check virtualenv pths are up-to-date r=ahal
It's possible for a virtualenv to have an incorrect list of directories
to add to the sys.path, such as the following cases:
* Its creation got cancelled halfway through
* The list of pths changed in a new revision
* It got modified by an external tool

By validating the list of provided pths against the list of required
pths, we can ensure that the virtualenv is more dependably up-to-date.

Differential Revision: https://phabricator.services.mozilla.com/D119686
2021-07-20 21:42:02 +00:00
Mitchell Hentges 6e4b9c1e3b Bug 1717645: Resolve nested virtualenv requirements up-front r=ahal
This simplifies consumer logic, since they get the parsed list of pypi
and pth requirements, as well as the list of input files that were
parsed.

One benefit of this simplification is that we no longer
recursively create VirtualenvManagers.

Note that mach_bootstrap cannot (yet) take advantage
of `ParseMachEnvRequirements` because of a dependency cycle:
* `mach_bootstrap` must set up the `sys.path` to import
  `ParseMachEnvRequirements`.
* `mach_bootstrap` would want `ParseMachEnvRequirements` to
  determine which paths to add to the `sys.path`.

Differential Revision: https://phabricator.services.mozilla.com/D119685
2021-07-20 21:42:01 +00:00
Barret Rennie b09836e3a7 Bug 1721472 - Correctly return whether or not we are in an SSH connection r=mhentges
Differential Revision: https://phabricator.services.mozilla.com/D120393
2021-07-20 21:13:17 +00:00
Brindusan Cristian 262e09aeea Backed out changeset 919c542f5b8b (bug 1720951) for causing build bustages and py3 failures in test_jarmaker.py.
CLOSED TREE
2021-07-20 19:02:16 +03:00
surajeet310 2f5fc1d2b8 Bug 1720951 - Removed instances of 'ensure_bytes()' from python/mozbuild/mozbuild r=mhentges
Inlined the usage of 'ensure_bytes()' and removed dependency of 'ensure_bytes()' from util.py .

Differential Revision: https://phabricator.services.mozilla.com/D120325
2021-07-20 15:15:18 +00:00
Sebastian Hengst 5a4f8ee0eb Backed out 13 changesets (bug 1709640, bug 1718341) for failing geckoview-junit tasks on shippable lite platform. CLOSED TREE
Backed out changeset abc4249b08a0 (bug 1709640)
Backed out changeset c7f2fa8d7535 (bug 1709640)
Backed out changeset 45b07d37f247 (bug 1709640)
Backed out changeset a15b3b87d71e (bug 1718341)
Backed out changeset a87639f2921b (bug 1718341)
Backed out changeset 31a57851c6d1 (bug 1718341)
Backed out changeset c7de56a1f886 (bug 1718341)
Backed out changeset 5e76806752c8 (bug 1718341)
Backed out changeset 74cfe6df218b (bug 1718341)
Backed out changeset 3bd0e376de51 (bug 1718341)
Backed out changeset 176238f92cfe (bug 1718341)
Backed out changeset 38d622f60fa5 (bug 1718341)
Backed out changeset 2e16fa5f0a1f (bug 1718341)
2021-07-20 11:31:01 +02:00
Agi Sferro 4881f826db Bug 1718341 - Download Android AVD from TL job artifact. r=nalexander
Differential Revision: https://phabricator.services.mozilla.com/D119542
2021-07-20 03:23:32 +00:00
Agi Sferro be84115bc0 Bug 1718341 - Update Android cmdtools to 4.0. r=nalexander,glandium
The SDK Tools version we're currently using is deprecated and was last released
in 2017. It's time for an upgrade!

Differential Revision: https://phabricator.services.mozilla.com/D119226
2021-07-20 03:23:32 +00:00
Agi Sferro fdce01f3f5 Bug 1718341 - Remove redundant override in MobileAndroidBootstrapper. r=nalexander
Differential Revision: https://phabricator.services.mozilla.com/D119998
2021-07-20 03:23:31 +00:00
Agi Sferro 2aca7bde2e Bug 1718341 - Prewarm AVD before packaging r=nalexander
The very first time an Android AVD starts it runs some one time jobs to
properly set up the AVD.

To avoid running the setup every time we run tests in automation, we can boot
the AVD before packaging it so that the testing jobs can use a "prewarmed" AVD
instead.

Differential Revision: https://phabricator.services.mozilla.com/D119225
2021-07-20 03:23:31 +00:00
Agi Sferro 43584d5056 Bug 1718341 - Generate Android AVD in TL job instead of using tooltool r=aklotz,nalexander
Tooltool images are hard to update because we don't provide a script to
generate the image and documentation is often inaccurate.

This patch makes it so we generate the AVD in the android-sdk TL job instead.

Differential Revision: https://phabricator.services.mozilla.com/D119221
2021-07-20 03:23:29 +00:00
surajeet310 d50cbbff03 Bug 1717651 - Removed 'ensure_subprocess_env()' from mozbuild/util.py r=mhentges
'ensure_subprocess_env()' is obsolete as Python 2 is no longer used in python/mozbuild .

Differential Revision: https://phabricator.services.mozilla.com/D120102
2021-07-19 21:22:27 +00:00
Alex Lopez 77ce415604 Bug 1696251 - Replace self with command_context where possible in existing mach commands. r=mhentges,webdriver-reviewers,perftest-reviewers,whimboo
This step removes all the dependencies of mach commands to
having a MachCommandBase as the `self` by using the `command_context`
argument instead. This also removes any remaining statefulness from those
classes that implement mach commands, ultimately making it easier to move
existing commands out of classes in a follow-up.

Differential Revision: https://phabricator.services.mozilla.com/D118058
2021-07-19 16:04:25 +00:00
Bob Owen b2d767fbb9 Bug 1716024 p2: Flag all binaries apart from firefox, xpcshell and plugin-container as CET compatible. r=glandium
We will only run the processes in CET compatible modules only mode when not
using the JIT code. So marking xul.dll as compatible should be OK.

Differential Revision: https://phabricator.services.mozilla.com/D117551
2021-07-19 07:36:19 +00:00
Mitchell Hentges 75de81c99a Bug 1720935: Move state_dir creation comment r=ahal
I'm guessing that the code moved to the bootstrap command but the
associated comment was forgotten and left abandoned, which is very sad.

Differential Revision: https://phabricator.services.mozilla.com/D120098
2021-07-16 21:47:53 +00:00
Mitchell Hentges 83721fed8d Bug 1720925: Move platform-specific Python upgrade instructions r=ahal
There's a "minimum Python check" that happens when Mach
bootstraps that makes the bootstrap command's check obsolete.

Since we want to use modern Python features throughout Mach, we need to
move Python-out-of-date logic up to the front of the process.

We're duplicating the OS-detection logic a bit, but now we regain
per-platform Python instructions.

Differential Revision: https://phabricator.services.mozilla.com/D120093
2021-07-16 21:47:52 +00:00
Mitchell Hentges 210af29ab0 Bug 1720925: Remove Python 2 warning from bootstrap r=ahal
We no longer require (or even allow) Python 2 for Mach, so the
"missing Python 2" warning is obsolete and just causes confusion.

Differential Revision: https://phabricator.services.mozilla.com/D120092
2021-07-16 21:47:52 +00:00
Butkovits Atila df5b6f3929 Backed out 13 changesets (bug 1709640, bug 1718341) for causing android build bustages. CLOSED TREE
Backed out changeset 20645ae32b84 (bug 1709640)
Backed out changeset 7c45682a8f98 (bug 1709640)
Backed out changeset 667e768fd75e (bug 1709640)
Backed out changeset f072adb45b85 (bug 1718341)
Backed out changeset cf443dddb5f5 (bug 1718341)
Backed out changeset 09d3d3e4ce7d (bug 1718341)
Backed out changeset 2b47cc469477 (bug 1718341)
Backed out changeset 8ec277361e46 (bug 1718341)
Backed out changeset b6c4c82d3efc (bug 1718341)
Backed out changeset 8afb8fcd0dc3 (bug 1718341)
Backed out changeset 7723d3a9ba7f (bug 1718341)
Backed out changeset f33b120a77d0 (bug 1718341)
Backed out changeset e7bf6deac270 (bug 1718341)
2021-07-17 01:02:51 +03:00
Butkovits Atila 28088b069e Backed out changeset b533ae02d367 (bug 1705420) as requested by Kimberly. CLOSED TREE 2021-07-16 23:22:23 +03:00
Agi Sferro 0c28adfdc3 Bug 1718341 - Download Android AVD from TL job artifact. r=nalexander
Differential Revision: https://phabricator.services.mozilla.com/D119542
2021-07-16 17:49:57 +00:00
Agi Sferro e63d6a91a2 Bug 1718341 - Update Android cmdtools to 4.0. r=nalexander,glandium
The SDK Tools version we're currently using is deprecated and was last released
in 2017. It's time for an upgrade!

Differential Revision: https://phabricator.services.mozilla.com/D119226
2021-07-16 17:49:56 +00:00
Agi Sferro b7965f3424 Bug 1718341 - Remove redundant override in MobileAndroidBootstrapper. r=nalexander
Differential Revision: https://phabricator.services.mozilla.com/D119998
2021-07-16 17:49:56 +00:00
Agi Sferro ee741ede76 Bug 1718341 - Prewarm AVD before packaging r=nalexander
The very first time an Android AVD starts it runs some one time jobs to
properly set up the AVD.

To avoid running the setup every time we run tests in automation, we can boot
the AVD before packaging it so that the testing jobs can use a "prewarmed" AVD
instead.

Differential Revision: https://phabricator.services.mozilla.com/D119225
2021-07-16 17:49:56 +00:00
Agi Sferro fa3a9445b3 Bug 1718341 - Generate Android AVD in TL job instead of using tooltool r=aklotz,nalexander
Tooltool images are hard to update because we don't provide a script to
generate the image and documentation is often inaccurate.

This patch makes it so we generate the AVD in the android-sdk TL job instead.

Differential Revision: https://phabricator.services.mozilla.com/D119221
2021-07-16 17:49:54 +00:00
Butkovits Atila a07f790e42 Backed out changeset e1921c5112d8 (bug 1696251) for causing bustages complaining about 'CommandContext'. CLOSED TREE 2021-07-16 20:35:55 +03:00
Mitchell Hentges 09c7c7c9dc Bug 1720591: Remove unused patch_main() for Python < 3.4 on Windows r=ahal
Now that we use Python 3.6+, this code is obsolete.

Differential Revision: https://phabricator.services.mozilla.com/D119929
2021-07-16 16:34:42 +00:00
Alex Lopez 190e03aaab Bug 1696251 - Replace self with command_context where possible in existing mach commands. r=mhentges,webdriver-reviewers,perftest-reviewers,whimboo
This step removes all the dependencies of mach commands to
having a MachCommandBase as the `self` by using the `command_context`
argument instead. This also removes any remaining statefulness from those
classes that implement mach commands, ultimately making it easier to move
existing commands out of classes in a follow-up.

Differential Revision: https://phabricator.services.mozilla.com/D118058
2021-07-16 15:51:29 +00:00
Mike Hommey 43e5bb336f Bug 1720822 - Don't install lucetc during mach bootstrap. r=firefox-build-system-reviewers,andi
At the moment, lucetc is not useful by default, only when enabling wasm
sandboxed libraries manually, and in that case, configure knows to
bootstrap lucetc itself. Furthermore, lucetc is set to be replaced with
wasm2c, which is in-tree and doesn't require bootstrapping at all.

Differential Revision: https://phabricator.services.mozilla.com/D120051
2021-07-16 08:36:26 +00:00
Sandor Molnar c40ce62c3c Backed out 10 changesets (bug 1718341) for causing wrench bustages. CLOSED TREE
Backed out changeset e5bd150a58b2 (bug 1718341)
Backed out changeset e82e752b8f44 (bug 1718341)
Backed out changeset d9462c727c8f (bug 1718341)
Backed out changeset 2bd647754780 (bug 1718341)
Backed out changeset 31dfce45985f (bug 1718341)
Backed out changeset 03266cea345d (bug 1718341)
Backed out changeset 3920b27acecb (bug 1718341)
Backed out changeset 2b17603c5e65 (bug 1718341)
Backed out changeset ebc0d5d4f537 (bug 1718341)
Backed out changeset 1c8844157b8d (bug 1718341)
2021-07-16 03:40:56 +03:00
Agi Sferro ea13a76521 Bug 1718341 - Download Android AVD from TL job artifact. r=nalexander
Differential Revision: https://phabricator.services.mozilla.com/D119542
2021-07-15 23:22:18 +00:00
Agi Sferro 3c257f8ca5 Bug 1718341 - Update Android cmdtools to 4.0. r=nalexander,glandium
The SDK Tools version we're currently using is deprecated and was last released
in 2017. It's time for an upgrade!

Differential Revision: https://phabricator.services.mozilla.com/D119226
2021-07-15 23:22:17 +00:00
Agi Sferro 36dcc7ce37 Bug 1718341 - Remove redundant override in MobileAndroidBootstrapper. r=nalexander
Differential Revision: https://phabricator.services.mozilla.com/D119998
2021-07-15 23:22:17 +00:00
Agi Sferro 9ee3064894 Bug 1718341 - Prewarm AVD before packaging r=nalexander
The very first time an Android AVD starts it runs some one time jobs to
properly set up the AVD.

To avoid running the setup every time we run tests in automation, we can boot
the AVD before packaging it so that the testing jobs can use a "prewarmed" AVD
instead.

Differential Revision: https://phabricator.services.mozilla.com/D119225
2021-07-15 23:22:17 +00:00
Agi Sferro 164fc2191d Bug 1718341 - Generate Android AVD in TL job instead of using tooltool r=aklotz,nalexander
Tooltool images are hard to update because we don't provide a script to
generate the image and documentation is often inaccurate.

This patch makes it so we generate the AVD in the android-sdk TL job instead.

Differential Revision: https://phabricator.services.mozilla.com/D119221
2021-07-15 23:22:15 +00:00
Cosmin Sabou 3cd7630738 Backed out changeset 2c1257a95130 (bug 1720591) for causing build bustages complaining about patch_main. CLOSED TREE 2021-07-16 02:49:45 +03:00
Mitchell Hentges b5d3b00394 Bug 1720591: Remove unused patch_main() for Python < 3.4 on Windows r=ahal
Now that we use Python 3.6+, this code is obsolete.

Differential Revision: https://phabricator.services.mozilla.com/D119929
2021-07-15 21:44:37 +00:00
surajeet310 ec79e93bd5 Bug 1717651 - Removed instances of 'ensure_subprocess_env' from python/mozversioncontrol r=mhentges
Since Python 2 is no longer used in python/mozversioncontrol, hence 'ensure_subprocess_env' is obsolete.

Differential Revision: https://phabricator.services.mozilla.com/D119905
2021-07-15 21:34:08 +00:00
amy churchwell 0c7d4a9357 Bug 1719727 - Change "Personalize" on New Tab to gear icon. r=prathiksha,fluent-reviewers,Gijs,thecount
Differential Revision: https://phabricator.services.mozilla.com/D119657
2021-07-15 19:36:50 +00:00
Anuj Gautam 557acb4135 Bug 1720761: Added kali to the list of debian configurations. r=mhentges
Differential Revision: https://phabricator.services.mozilla.com/D120003
2021-07-15 19:06:10 +00:00
Mike Hommey 77f0b36028 Bug 1720595 - Don't install dump_syms during mach bootstrap. r=firefox-build-system-reviewers,nalexander
Bug 1719229 broke the install by accident, but configure takes care of
it by default since bug 1716911, bug 1716912 or bug 1717585 depending on
the platform, so we can just stop installing it from mach bootstrap.

Differential Revision: https://phabricator.services.mozilla.com/D119933
2021-07-15 04:47:10 +00:00
surajeet310 1903d302ab Bug 1717651 - Removed instances of 'ensure_subprocess_env' from python/mozperftest r=mhentges
Since Python 2 is no longer used in python/mozperftest, hence 'ensure_subprocess_env' is obsolete.

Differential Revision: https://phabricator.services.mozilla.com/D119898
2021-07-14 16:33:46 +00:00
surajeet310 88365fbe52 Bug 1717651 - Removed instances of 'ensure_subprocess_env' from python/mozbuild r=mhentges
Since Python 2 is no longer used in python/mozbuild, hence 'ensure_subprocess_env' is obsolete.

Differential Revision: https://phabricator.services.mozilla.com/D119791
2021-07-14 15:17:27 +00:00
Mike Hommey 487119e349 Bug 1719229 - Allow the build system to use a sysroot for the host part of the build. r=firefox-build-system-reviewers,andi
Differential Revision: https://phabricator.services.mozilla.com/D119847
2021-07-14 08:54:43 +00:00
Jonathan Kew 0132f4c4df Bug 1720308 - Support VSCode installed as a snap package on Linux. r=firefox-build-system-reviewers,nalexander
Differential Revision: https://phabricator.services.mozilla.com/D119769
2021-07-13 15:04:06 +00:00
Kimberly Sereduck 210d1cc333 Bug 1705420: Report the version of the application being tested in our perfherder data r=perftest-reviewers,sparky
implementing get_browser_meta for mozperftest

Differential Revision: https://phabricator.services.mozilla.com/D113861
2021-07-13 12:13:38 +00:00
surajeet310 f2e1419d77 Bug 1717651 - Removed instances of 'ensure_subprocess_env' from python/mach r=mhentges
Since Python 2 is no longer used in python/mach, hence 'ensure_subprocess_env' is obsolete .

Differential Revision: https://phabricator.services.mozilla.com/D119185
2021-07-12 21:38:13 +00:00
Mitchell Hentges 1256637095 Bug 1720113: Temporarily add "python" to default clobber set r=firefox-build-system-reviewers,nalexander
Until virtualenvs are more dependably consistent, we should purge them
by default when "./mach clobber" is run.

Differential Revision: https://phabricator.services.mozilla.com/D119651
2021-07-12 16:43:09 +00:00
Florin Strugariu 2b3375937c Bug 1715478 - Automated recording for desktop with a list sites r=sparky
Differential Revision: https://phabricator.services.mozilla.com/D117298
2021-07-09 07:12:27 +00:00
june wilde 7db7c6efaf Bug 1716398 - support tracking tagged releases in mach vendor; r=tjr
Differential Revision: https://phabricator.services.mozilla.com/D118422
2021-07-08 18:20:04 +00:00
Mitchell Hentges f20cdb0c12 Bug 1712804: Check if `comm` directory is empty in TB identification r=ahal
Some developers hold onto a `comm` directory as a mount point
that they conditionally populate depending on whether they're working on
Firefox or Thunderbird.
The "`comm` directory exists == is a thunderbird checkout"
assumption isn't compatible with this workflow.

The fix embraced by this patch is to check if the `comm` directory has
any contents.

Differential Revision: https://phabricator.services.mozilla.com/D119193
2021-07-06 18:33:58 +00:00
Emilio Cobos Álvarez 780f3a05d2 Bug 1719144 - Fix various imports for Python 3.10. r=firefox-build-system-reviewers,mhentges
These are enough for me to run bootstrap+configure+build.

Some touch third-party code (gyp), but per discussion in the earlier
versions of this revision that seems fine.

Differential Revision: https://phabricator.services.mozilla.com/D119080
2021-07-06 15:26:29 +00:00
Mike Hommey 544436ed03 Bug 1718156 - Avoid some configure tests creating the common virtualenv. r=firefox-build-system-reviewers,mhentges,andi
When running `mach python-test` on a clobbered tree, we create both the
python-test and common virtualenvs, but the former should be enough. The
latter is only created when running tests that use the BaseConfigureTest
class, when it indirectly includes init.configure, which makes the
configure sandbox run the virtualenv_python3 function, which ends up
initializing the common virtualenv. We only don't end up re-executing
that virtualenv python thanks to 6680ca0acc27 which was a workaround
that would allow any virtualenv in $objdir/_virtualenvs, while still
creating a virtualenv it won't use.

Tests now tell virtualenv_python3 to not use the normal virtualenv
manager, and remove the workaround.

Differential Revision: https://phabricator.services.mozilla.com/D118775
2021-07-05 22:33:56 +00:00
Mitchell Hentges c25aec1f74 Bug 1717104: Update "pth" docs for virtualenvs r=ahal
The "pth" action no longer has a customizable filename, and the docs
should be updated accordingly.

Depends on D118609

Differential Revision: https://phabricator.services.mozilla.com/D118610
2021-07-05 22:10:35 +00:00
Mitchell Hentges 86ae5fbc84 Bug 1717104: Remove obsolete "set-variable" notes from virtualenv docs r=ahal
Since https://phabricator.services.mozilla.com/D115921, "set-variable"
is no longer a valid action. Looks like that patch forgot to remove the
associated docs

Differential Revision: https://phabricator.services.mozilla.com/D118609
2021-07-05 22:10:35 +00:00
surajeet310 498013ddc6 Bug 1717651 - Removed instances of 'ensure_subprocess_env()' from virtualenv.py r=mhentges
Since Python 2 is no longer used in virtualenv.py ,hence 'ensure_subprocess_env()' is obsolete.

Differential Revision: https://phabricator.services.mozilla.com/D118883
2021-07-05 19:59:53 +00:00
Sandor Molnar fe63cb08df Backed out 2 changesets (bug 1719144) for causing build bustages. CLOSED TREE
Backed out changeset 8e4b00b9d57c (bug 1719144)
Backed out changeset d2a88c5423a0 (bug 1719144)
2021-07-05 21:15:50 +03:00
Emilio Cobos Álvarez 52a85aaca1 Bug 1719144 - Fix various imports for Python 3.10. r=firefox-build-system-reviewers,mhentges
These are enough for me to run bootstrap+configure+build.

Some touch third-party code (gyp), but per discussion in the earlier
versions of this revision that seems fine.

Differential Revision: https://phabricator.services.mozilla.com/D119080
2021-07-05 17:55:41 +00:00
Mitchell Hentges 9ee9a69ade Bug 1717905: M1 Macs don't support Android targets r=glandium
Also filter out bootstrapping for Android targets.

Differential Revision: https://phabricator.services.mozilla.com/D118727
2021-07-05 17:44:20 +00:00
Mitchell Hentges 733d1bb939 Bug 1717905: Fix M1 Mac bootstrap choice offset r=glandium
When we customized the list of possible `applications` for M1 Macs, the
change was only applying to the displayed list, but not the actual
list that the choice was indexed against.

This is fixed by properly using `applications` instead of
`APPLICATIONS` where relevant.

Note that it's still possible to manually request an
`artifact_mode` if you provide it as an argument for `./mach bootstrap`,
but this workflow is usually only done by devs who are familiar with all
the applications, and therefore likely know what they're doing.

Differential Revision: https://phabricator.services.mozilla.com/D118625
2021-07-05 17:44:19 +00:00
june wilde 68adbeeae5 Bug 1703066 - add tracking key to mozyaml file schema; r=tjr
Differential Revision: https://phabricator.services.mozilla.com/D110829
2021-06-29 14:01:58 +00:00
Mike Hommey 81228f00cd Bug 1717947 - Remove packager support for brotli in Jars. r=firefox-build-system-reviewers,nalexander
This was added ing bug 1352595. It has only been used briefly, and we
since then have gone all the way to never compress omni.ja. Even if we
did go back to compressing, we'd probably go with zstd rather than
brotli.
The gecko-side support for this has only ever been nightly-only.

Differential Revision: https://phabricator.services.mozilla.com/D118655
2021-06-25 00:20:10 +00:00
Mitchell Hentges b7dde05a8d Bug 1717786: Update mozperfest runner's list of in-tree Python packages r=kimberlythegeek
After replacing a bunch of vendored source packages with wheels, the
paths to the raw source changed.

Differential Revision: https://phabricator.services.mozilla.com/D118723
2021-06-24 18:02:53 +00:00
Mitchell Hentges 43949e4a1c Bug 1717702: Resolve `print-checks` python unpacking typo r=andi
`_get_infer_config()` returns 3 items, but the existing code expects 2.
Add a third `_` to capture the other unwanted item.

Differential Revision: https://phabricator.services.mozilla.com/D118535
2021-06-24 13:21:57 +00:00
Alexander J. Vincent b12c2e5817 Bug 1717650 - Support RockyLinux 8.4. r=mhentges
Differential Revision: https://phabricator.services.mozilla.com/D118539
2021-06-22 20:34:13 +00:00
Alexander J. Vincent 2c7ff94a69 Bug 1717650 - Drop requirement for wireless-tools-devel package. r=mhentges
wireless-tools-devel isn't available on Rocky Linux. Fortunately, it doesn't appear to be needed for the build anymore, and it was added almost a decade ago, so it should be safe to remove.

Differential Revision: https://phabricator.services.mozilla.com/D118532
2021-06-22 20:34:13 +00:00
Tarik Eshaq 453be8c6fd Bug 1716450 - Removes fxa-client rust code from Desktop. r=rfkelly
Differential Revision: https://phabricator.services.mozilla.com/D118238
2021-06-22 17:04:35 +00:00
Marian-Vasile Laza 4728ef73b2 Bug 1700534 - black lint fix r=fix CLOSED TREE 2021-06-22 09:22:33 +03:00
Mike Hommey 744db845c6 Bug 1700534 - Coalesce RLBox wasmboxed libraries. r=firefox-build-system-reviewers,shravanrn,bholley,andi,mhentges
Differential Revision: https://phabricator.services.mozilla.com/D116440
2021-06-22 05:31:33 +00:00
Mitchell Hentges ee5a948268 Bug 1716600: Don't use node/npm that are too old or have no version r=firefox-build-system-reviewers,nalexander
In the existing behaviour, we would return a node/npm executable from
`find_executable()`, even if we couldn't extract a version from it or if
the extracted version was insufficient.

Now, we fail early if the detected node/npm is insufficient.

Differential Revision: https://phabricator.services.mozilla.com/D118377
2021-06-21 16:47:56 +00:00
Mitchell Hentges 00d96c3fed Bug 1716600: Remove looping from Node's find_executable() r=firefox-build-system-reviewers,nalexander
Since we're no longer checking for both "nodejs" and "node",
`find_executable()` can be simplified.

Differential Revision: https://phabricator.services.mozilla.com/D118376
2021-06-21 16:47:55 +00:00
Mitchell Hentges 73814a7f0f Bug 1716600: Always search for "node" instead of "nodejs" r=firefox-build-system-reviewers,nalexander
As :julienw beautifully described on Bugzilla,
we no longer need to search for "nodejs" since all
packages providing Node >= v12 will provide the `node`
binary.
See his comment here:
https://bugzilla.mozilla.org/show_bug.cgi?id=1716600#c4

Base-toolchain tasks with Node v10 should still be
happy because, unlike Debian's Node v10, they provide
a `node` binary instead of `nodejs`.

Differential Revision: https://phabricator.services.mozilla.com/D118375
2021-06-21 16:47:55 +00:00
Aki Sasaki 5b64d5df25 Bug 1659691 - Update cross-channel algorithm to not replay history. r=firefox-build-system-reviewers,releng-reviewers,bhearsum,mhentges
(Tl;dr: The original author of this patch was axel@mozilla.com (:pike).
This patch has been modified to run in production automation.)

Cross-channel takes all the en-US strings from the shipping train branches
and consolidates them into a single repository. Originally, this ran out of
https://hg.mozilla.org/users/axel_mozilla.com/cross-channel-experimental/ .
The original version was coupled tightly with specific hg internals, making
hg upgrades fragile. Axel wrote
https://bug1659691.bmoattachments.org/attachment.cgi?id=9170636 before
handing off project ownership; this is the original patch to stop replaying
history to simplify the logic.

This patch also automates the previously manual preparation steps, allows
for running both in automation and locally, and adds retries for production
robustness.

Differential Revision: https://phabricator.services.mozilla.com/D116537
2021-06-17 18:52:58 +00:00
Mitchell Hentges 2438ea8889 Bug 1712133: Validate that pypi dependencies are installed r=ahal
Check that all expected pypi packages are installed
when checking if a virtualenv is up-to-date.

Differential Revision: https://phabricator.services.mozilla.com/D117712
2021-06-17 14:58:17 +00:00
Mitchell Hentges 2fa59acd4f Bug 1712133: Adds "pypi" action to virtualenv `handle_package()` r=ahal
The `pypi` action uses `pip` to fetch a package and its dependencies

Differential Revision: https://phabricator.services.mozilla.com/D115925
2021-06-17 14:58:17 +00:00
Mitchell Hentges 24f634d924 Bug 1712133: Remove "pth" name customization r=ahal
Having separate `<name>.pth` files in the virtual environments
isn't providing an advantage. We can simplify configuration
by putting all `pth` adjustments into a single file: `mach.pth`.

Differential Revision: https://phabricator.services.mozilla.com/D117710
2021-06-17 14:58:16 +00:00
Mitchell Hentges 31474e8925 Bug 1712133: Make virtualenv package parsing more specific r=ahal
This has two benefits:
1. `handle_package()` becomes more clear - rather than referring to
   `action` and `package` with array index numbers, we now give
   them real names. The benefit here is also shown in `up_to_date()`.
2. This makes the top-level parser for `packages()` less opinionated
   about sub-formats: if an action has a nested structure, it should
   have the flexibility to define what it looks like.

Differential Revision: https://phabricator.services.mozilla.com/D117708
2021-06-17 14:58:16 +00:00
Mitchell Hentges 722ac646db Bug 1713857: Don't set PYTHONEXECUTABLE environment variable r=glandium
After some testing in `try` and locally, the manual
`PYTHONEXECUTABLE` definitions shouldn't be needed
anymore.

There's been some work on Brew's
Python to improve its `sitecustomize` behaviour.
The most likely improvement is:
https://github.com/Homebrew/homebrew-core/pull/65297

However, I'm not fully confident in this change.
If it fails, it's more likely to affect developers
than CI. I think it's worth attempting a landing,
because if the variable is indeed obsolete, then
deleting it will avoid some spicy "action-at-a-
distance" behaviour.

Differential Revision: https://phabricator.services.mozilla.com/D117452
2021-06-17 13:28:29 +00:00
Marian-Vasile Laza 19f23a7e8b Bug 1716911 - black lint fix r=fix CLOSED TREE 2021-06-17 09:39:54 +03:00
Mike Hommey 83c4b6a318 Bug 1716911 - Switch --enable-bootstrap on by default on macos builds on central. r=firefox-build-system-reviewers,nalexander
Differential Revision: https://phabricator.services.mozilla.com/D118107
2021-06-17 04:59:42 +00:00
Mitchell Hentges 6d154c1ed3 Bug 1713377: Change vendoring to use wheels where possible r=ahal,glandium
Vendoring wheels has three benefits:
* There's far less files, so Firefox checkouts will be smaller.
* It works around `zipp` not allowing `pip install`
  from extracted source `tar.gz` files. Now, we should
  be able to use the pip resolver against vendored
  packages, which will be needed for future
  mach virtualenv work.
* `./mach vendor python` takes far less time to execute.

Since we need the raw Python to be available to add to the `sys.path`,
we extract the wheels before putting them in tree.
Due to the structure of some wheels being less nested
than of a source `tar.gz`, `common_virtualenv_packages`
needed to be adjusted accordingly.

`install_pip_package()` had to be tweaked as well since you can't
`pip install` an extracted wheel. So, we "re-bundle" the wheel
before installing from a vendored package.

Replace python packages with wheels where possible

This contains the vendoring changes caused by the
last patch.

For reviewing, there's a couple things to note:
* A bunch of files are deleted, since there's generally
  less files in a wheel than in a source archive.
* There's a new `.dist-info` directory for each
  extracted wheel, so expect roughly 5 or
  6 new files for each wheel'd package.
* There should be no source code changes other than
  moves from package names changing from having
  `-` to having `_`.

Differential Revision: https://phabricator.services.mozilla.com/D116512
2021-06-16 15:53:16 +00:00
Mitchell Hentges c335e37407 Bug 1690870: Remove "artifact mode" from bootstrap options on M1 Macs r=firefox-build-system-reviewers,andi
As pointed out by Harry, this will help in the short-term with
onboarding new developers who will be using M1 macs.

Note that this patch doesn't leverage the "instance" classes to define
whether they're artifact-mode compatible or not, and that's because:
* Most (all except one?) of the systems support artifact mode, therefore
* Since this is a temporary workaround, it's more deletable to have
  a top-level `if/else` than to add a `supports_artifact_mode` property
  to each Bootstrapper.

Differential Revision: https://phabricator.services.mozilla.com/D117946
2021-06-16 14:31:07 +00:00
surajeet310 c3511254e0 Bug 1714688 - Removed 'if PY2'/'if PY3' logic from python/mozbuild r=mhentges
Python2 is no longer used in Mozbuild.

Differential Revision: https://phabricator.services.mozilla.com/D117561
2021-06-15 19:31:21 +00:00
Gregory Mierzwinski a5873d50c0 Bug 1716378 - Use correct test modules path in xpcshell layer. r=perftest-reviewers,kimberlythegeek
Differential Revision: https://phabricator.services.mozilla.com/D117690
2021-06-15 17:01:58 +00:00
Mitchell Hentges 9f7c330b60 Bug 1712133: Remove "windows" action support from virtualenvs r=ahal
Last year, we stopped vendoring Python packages that have native
code. Since we have only had pure-python packages since, the
Windows-specific qualifier (or excluder in the case of `!windows`)
hasn't been needed.

I don't foresee us needing it again, but if anything we can peel it
back from `hg` history if this assumption is incorrect.

Differential Revision: https://phabricator.services.mozilla.com/D117468
2021-06-15 14:46:55 +00:00
Mitchell Hentges 1b74e5a575 Bug 1713201: Resolve test_telemetry failing locally r=ahal
On Windows, Python's temporary directories are placed in `%APPDATA%`.
Unfortunately, the failing tests want to assume that some "other"
path (as a subpath of a tempdir) won't be within any
`filter_args()` known top-level paths, which includes the user's home
dir.

The only reason that this assumption has passed in CI so far is due
to Windows case insensitivity: running locally had `~` evaluated
to `c:\Users\...`, while running in CI had it be `c:\users\...`.
Since the tempdir path started with `C:\Users`, `filter_args()`
would `<path omitted>` in CI but would `$HOME/...` locally.

By not basing the `other_path` on `tempdir`, we now consistently get the
same results across platforms.

Differential Revision: https://phabricator.services.mozilla.com/D117698
2021-06-15 14:45:53 +00:00
Alexandru Michis 542a65a629 Bug 1715282 - Fix black py lint in test_toolchain_configure.py a=lint-fix
CLOSED TREE
2021-06-15 18:04:43 +03:00
Mike Hommey 7f9a51a8f5 Bug 1715282 - Bump minimum Rust version to 1.51. r=firefox-build-system-reviewers,andi
Also update the estimated rust versions for future releases, based on
the updated Firefox release dates.

Differential Revision: https://phabricator.services.mozilla.com/D117737
2021-06-15 09:23:20 +00:00
Iulian Moraru 952b525e6f Backed out changeset 55a7556ae185 (bug 1715282) for causing base-toolchain bustages and python failures on test_toolchain_configure.py. CLOSED TREE 2021-06-15 10:46:39 +03:00
Jeff Muizelaar e5c5ce020a Bug 1715282 - Bump minimum Rust version to 1.50. r=glandium
Various people want 1.50 and it's been out for a while so let's require it.

Differential Revision: https://phabricator.services.mozilla.com/D117158
2021-06-15 05:46:02 +00:00
surajeet310 ec19bcfbf2 Bug 1714688 - Removed 'if PY2' logic from python/mozrelease r=mhentges
Python2 is no longer used in Mozrelease

Differential Revision: https://phabricator.services.mozilla.com/D117583
2021-06-14 20:41:31 +00:00
Csoregi Natalia 6b0106d5b1 Backed out changeset 15c6a16f5828 (bug 1714688) for docker images bustage. CLOSED TREE 2021-06-14 20:51:29 +03:00
Csoregi Natalia 5136d2de85 Bug 1714684 - Fix linting. r=fix CLOSED TREE 2021-06-14 20:47:59 +03:00
surajeet310 0ac7b98b51 Bug 1714688 - Removed 'if PY2'/'if PY3' logic from python/mozbuild r=mhentges
Python2 is no longer used in Mozbuild.

Differential Revision: https://phabricator.services.mozilla.com/D117561
2021-06-14 16:10:11 +00:00
Mitchell Hentges b685c830d4 Bug 1714684: Resolve `assert_has_calls()` usage error r=ahal
The `calls` parameter is expected to be an iterable
container of calls, not a singular call.

This was working in `mock-1.0.0` because `calls`
was (confusingly) allowed to be a single item
if `any_order==False`. This behaviour isn't
the same in the standard library.

Differential Revision: https://phabricator.services.mozilla.com/D117074
2021-06-14 15:34:47 +00:00
Mitchell Hentges e389cc4cc4 Bug 1714684: Remove usages of vendored "mock" library r=perftest-reviewers,ahal,sparky
Python 3 has access to `unittest.mock` in the standard library.

Differential Revision: https://phabricator.services.mozilla.com/D117073
2021-06-14 15:34:46 +00:00
Butkovits Atila ce6e97604b Backed out changeset 7648560f810d (bug 1715282) for causing build bustages. CLOSED TREE 2021-06-11 22:47:34 +03:00
Jeff Muizelaar 5ca0d5fb13 Bug 1715282 - Bump minimum Rust version to 1.50. r=glandium
Various people want 1.50 and it's been out for a while so let's require it.

Differential Revision: https://phabricator.services.mozilla.com/D117158
2021-06-11 19:27:22 +00:00
surajeet310 6ea83c8ef9 Bug 1714788 - Fixed more Sphinx warnings in 'mach doc' r=sylvestre DONTBUILD
Differential Revision: https://phabricator.services.mozilla.com/D117419
2021-06-10 19:33:53 +00:00
Mitchell Hentges dc2fa7819c Bug 1713610: Require PyPI-vendored packages be added to requirements.in r=ahal
`./mach vendor python <package>` was already adding its new package
to `requirements.txt`, so we were getting full resolver support, which
is good. However, it caused our `requirements.in` file to start
getting out-of-date, and therefore made it harder to identify the
top-level dependencies.

Arguably, we could have `./mach vendor python <package>` automatically
update `requirements.in`, too, but then we need to solve the edge cases,
such as "What if the package is already in `requirements.in`? What if
that existing item has a different version?"

The hardest part of updating `requirements.in` was finding it, so I've
also modified the `./mach vendor python` help text to make it more
identifiable.

Differential Revision: https://phabricator.services.mozilla.com/D116386
2021-06-10 19:19:06 +00:00
Nick Alexander bef7042147 Bug 1524662 - Add --enable-path-remapping producing compiled objects with generic paths. r=glandium
For "c" (i.e., gcc, clang, and clang-cl), this configures the
`-f{debug,macro}-path-prefix` flags.  We'd prefer to use
`-ffile-path-prefix`, but it seems that `clang-cl` does not recognize
that flag.

For "rust" (i.e., rustc/cargo), this configures `--remap-path-prefix`.

This is one step toward getting `sccache` hits across source and
object directories.

Differential Revision: https://phabricator.services.mozilla.com/D113065
2021-06-10 17:08:06 +00:00
Mike Conley f6ae5b2c2e Bug 1682022 - Convert to Fluent and / or fork the remainder of the AppMenu strings for sentence casing. r=flod,fluent-reviewers
Differential Revision: https://phabricator.services.mozilla.com/D117067
2021-06-09 18:01:24 +00:00