Sometimes tools install pypi at runtime via mach (e.g self.install_pip_package
/ self.install_pip_requirements). It's difficult to test these modules with
pytest because we usually won't be going through mach.
This gives tests the ability to depend on external pypi packages the same way
they might get installed when running via mach.
Note, I only added support for requirements.txt here because
python/mozbuild/mozbuild/virtualenv.py's 'install_pip_package' function is
completely busted with modern pip. And the pip used with |mach python-test| is
more modern than the one used with the regular build venv due to pipenv. We'll
need to fix this eventually, but that's another bug for another day.
Differential Revision: https://phabricator.services.mozilla.com/D22784
--HG--
extra : moz-landing-system : lando
Bug 1528123 added a way to specify a task id to influence mach
artifact. In the case of upcoming EME-enabled win64-aarch64 builds, we
need to be able to specify two different task ids, which a single
environment variable doesn't really allow.
So extend the scheme to check environment variables with the build
type included in the variable name.
Differential Revision: https://phabricator.services.mozilla.com/D23101
--HG--
extra : moz-landing-system : lando
Instead of over-estimating the number of items in a batch, do the opposite:
slightly under-estimate the number of items, then dispatch outstanding items
(by just adding one item to each batch).
Differential Revision: https://phabricator.services.mozilla.com/D23141
--HG--
extra : moz-landing-system : lando
Parts of this patch were taken from the original work of :bbouvier in Bug 1521772.
We needed to revert Bug 1521772 since it broken Windows compatibility.
Differential Revision: https://phabricator.services.mozilla.com/D23100
--HG--
extra : moz-landing-system : lando
Flake8 ignores the 'exclude' section of the .flake8.yml if you pass in a direct
path to a file. To get around this we have some custom logic to handle these
exclusions for us, but this custom logic didn't account for globs.
Differential Revision: https://phabricator.services.mozilla.com/D23145
--HG--
extra : moz-landing-system : lando
It's occasionally useful to know what versions are being discovered for
these variables. It's also convenient for logs from automation to
contain all the relevant information, rather than setting up a loaner
and running all the commands yourself.
Differential Revision: https://phabricator.services.mozilla.com/D22385
--HG--
extra : moz-landing-system : lando
The jar log is used for optimization of the packaged jar files according
to their usage patterns during a profile run. The current content of the
file currently come with 2 caveats:
- it contains entries for jar archives that aren't relevant to
packaging, which is not a problem in itself, but see below.
- it contains full paths for jar archives that may not correspond to the
location of the packaged directory (on e.g. Android, where the build
almost certainly doesn't happen in the same directory on the host as
Fennec runs in the emulator/on the device).
The current JarLog code does somehow handle the various ways paths are
currently presented, but it's clearly missing code to map the paths in
the log to packaged paths. Instead of requiring manual work and extra
build options to handle this mapping, and considering the caveats above,
it's just simpler to log archive paths as if they were relative to the
packaged application directory in a build, and use that during
packaging.
Depends on D21655
Differential Revision: https://phabricator.services.mozilla.com/D21656
--HG--
extra : moz-landing-system : lando
The jar log is used for optimization of the packaged jar files according
to their usage patterns during a profile run. The current content of the
file currently come with 2 caveats:
- it contains entries for jar archives that aren't relevant to
packaging, which is not a problem in itself, but see below.
- it contains full paths for jar archives that may not correspond to the
location of the packaged directory (on e.g. Android, where the build
almost certainly doesn't happen in the same directory on the host as
Fennec runs in the emulator/on the device).
The current JarLog code does somehow handle the various ways paths are
currently presented, but it's clearly missing code to map the paths in
the log to packaged paths. Instead of requiring manual work and extra
build options to handle this mapping, and considering the caveats above,
it's just simpler to log archive paths as if they were relative to the
packaged application directory in a build, and use that during
packaging.
Depends on D21655
Differential Revision: https://phabricator.services.mozilla.com/D21656
--HG--
extra : moz-landing-system : lando
Before this change linting python/mozbuild brought about many errors. Some of this errors could be fixed using |mach lint --fix|. The remaining errors where fixed in this bug.
Differential Revision: https://phabricator.services.mozilla.com/D20138
--HG--
extra : rebase_source : 8bd3d622d2221b981f08b8a080c1198b002cdc49
extra : amend_source : 3f6a70c1b7104a7c2d83e11fe911942771e331ea
Fennec has a value of OMNIJAR_NAME that contains a directory, contrary
to other platforms, and relies in post-packaging, pre-unpacking steps to
accommodate with the difference.
With this change, we just make the packaging and unpacking steps aware
of this setup, and make allow them to pack/unpack resources in foo/
under foo/$OMNIJAR_NAME, whether $OMNIJAR_NAME is a file name or a path.
This will, further down the road, allow the packager code to handle jar
logs from PGO instrumentation without munging them.
Differential Revision: https://phabricator.services.mozilla.com/D21654
--HG--
extra : moz-landing-system : lando
The underlying backends only support directories in LOCAL_INCLUDES
(since these ultimately translate to -I arguments to the compiler). We
should disallow filenames here, in case a developer accidentally
specifies a header file instead of a directory.
Differential Revision: https://phabricator.services.mozilla.com/D21698
--HG--
extra : moz-landing-system : lando
After landing bug 1515004, it is unnecessary to use `--with-android-sdk` when
developer uses default path of Android SDK by `./mach bootstrap`.
So let's remove `--with-android-sdk` from mozconfig example using
`./mach bootstrap`.
Differential Revision: https://phabricator.services.mozilla.com/D20601
--HG--
extra : moz-landing-system : lando
In cases like those in the added unit test, explicit options on the
command line could end up being silently ignored. So instead of that happening,
error out. Unfortunately, the error message is not entirely accurate,
but it's better than nothing. It's rare anyways (I only stumbled upon it
because I was trying to do something fishy), and correlation between the
error message and the corresponding changes should make it clear what's
going on.
Depends on D20822
Differential Revision: https://phabricator.services.mozilla.com/D20823
--HG--
extra : moz-landing-system : lando
Currently, when a dependency loop involve imply_option, it is possible
to end up with an error message saying the implied option is unknown,
when it fact it is. So instead of bailing out with a weird error
message, try to make things work (if the implied value is not different
from what's known), or bail with a more accurate message.
Differential Revision: https://phabricator.services.mozilla.com/D20822
--HG--
extra : moz-landing-system : lando
Optimizing jars without preloading/reordering data only moves the
jar central directory to the beginning of the file, which, without
preloading information, is not very useful. Let's just stop doing it if
there's not going to be preloading/reordering information at all.
Differential Revision: https://phabricator.services.mozilla.com/D21170
--HG--
extra : moz-landing-system : lando
On Android we apparently try to include the cpufeatures.c file from the
NDK as part of the generated-sources tarball. While this works, it makes
tar drop the bogus entry for safety reasons when unpacking the archive.
It also makes tar return an error, which is undesirable for searchfox.
It's better to just skip the entry when building the tarball.
Differential Revision: https://phabricator.services.mozilla.com/D20942
--HG--
extra : moz-landing-system : lando
For mach commands that have 'pass_context=True', we should implicitly add the
handler instance to the context. This will give mach command implementations an
easy way to access things like the command/subcommand names, the parser, argv
list, etc.
Differential Revision: https://phabricator.services.mozilla.com/D20521
--HG--
extra : moz-landing-system : lando
This patch adds detection for when icecream is in use to build telemetry.
icecream is commonly enabled in two ways: by either setting CC/CXX to point
to icecream's cc/c++ symlinks, or by setting adding
mk_add_options 'export CCACHE_PREFIX=icecc' to a mozconfig when also using
ccache. For the former, this patch adds a simple configure check to see
if CXX is a symlink to a file named 'icecc'. For the latter this patch adds
CCACHE_PREFIX as a configure subst to capture the value.
We don't currently have a facility for writing telemetry tests that depend on
configure values. Local manual testing shows that it does work as expected.
Differential Revision: https://phabricator.services.mozilla.com/D18138
--HG--
extra : moz-landing-system : lando
The motivations for this are:
1) Apply global excludes. This merges the exclusion rules defined in
tools/lint/mach_commands.py with the ones in .flake8.
2) Improve performance. Switching to a blacklist will result in a much longer
runtime for linting the entire tree and flake8 handles exclusions incredibly
slowly. Without this patch (and the blacklist change applied), I gave up
waiting after 30 minutes. With this patch, it takes 30 seconds.
Depends on D20495
Differential Revision: https://phabricator.services.mozilla.com/D20496
--HG--
rename : tools/lint/test/files/flake8/bad.py => tools/lint/test/files/flake8/subdir/exclude/bad.py
extra : moz-landing-system : lando
This will be re-used by the flake8 linter, so moving it into mozlint for
re-useability.
Depends on D20493
Differential Revision: https://phabricator.services.mozilla.com/D20494
--HG--
extra : moz-landing-system : lando
Coroiu Cristina
Alvaro Serero
Mike Hommey
Dustin J. Mitchell
Andrew Halberstadt
Gurzau Raul
Nicholas Cowles
Benjamin Bouvier
Aki Sasaki
Andi-Bogdan Postelnicu
Thomas Daede
Sylvestre Ledru
Nathan Froyd
Jason Laster
Brindusan Cristian
David Walsh
Axel Hecht
kerlyn
Mike Shal
Sebastian Hengst
Cosmin Sabou
Daniel Varga
Makoto Kato
Kartikaya Gupta
Panos Astithas
Ted Mielczarek
Christian Frey