This undoes the mistaken move of this directory into third_party that happened
as part of bug 1346025.
The .flake8 changes are required because lldbutils used to be excluded from
linting by virtue of being inside third_party, but would otherwise be subject
to it after this move. Fixing the existing lint errors will happen in:
- https://bugzilla.mozilla.org/show_bug.cgi?id=1605144
- https://bugzilla.mozilla.org/show_bug.cgi?id=1605145
Differential Revision: https://phabricator.services.mozilla.com/D57141
--HG--
rename : third_party/python/lldbutils/README.txt => python/lldbutils/README.txt
rename : third_party/python/lldbutils/lldbutils/__init__.py => python/lldbutils/lldbutils/__init__.py
rename : third_party/python/lldbutils/lldbutils/content.py => python/lldbutils/lldbutils/content.py
rename : third_party/python/lldbutils/lldbutils/general.py => python/lldbutils/lldbutils/general.py
rename : third_party/python/lldbutils/lldbutils/gfx.py => python/lldbutils/lldbutils/gfx.py
rename : third_party/python/lldbutils/lldbutils/layout.py => python/lldbutils/lldbutils/layout.py
rename : third_party/python/lldbutils/lldbutils/utils.py => python/lldbutils/lldbutils/utils.py
extra : moz-landing-system : lando
We recently updated the version of pip we use, and it no longer supports
--use-wheels.
Differential Revision: https://phabricator.services.mozilla.com/D57404
--HG--
extra : moz-landing-system : lando
This patch causes the newest version of NodeJS 8.17.0 to be used to build the mozilla toolchains installed on the CI machines as well as by `mach bootstrap`, which in turn gets an important security fix by updating to npm 6.13.4.
This patch also causes mach commands which call npm to abort if the version in use is older than 6.13.4 to avoid (especially) exploits on local developers that exfiltrate their data.
Differential Revision: https://phabricator.services.mozilla.com/D57674
--HG--
extra : moz-landing-system : lando
If mercurial is already installed via pip (or possibly other means),
then mach bootstrap will fail. We should check if we already have a
sufficient mercurial before forcing it to be installed with
ports/homebrew.
Differential Revision: https://phabricator.services.mozilla.com/D57566
--HG--
extra : moz-landing-system : lando
Currently BuildReader._relevant_mozbuild_finder attempted to exclude objdirs by
ignoring the pattern 'obj*'. However this can cause problems for developers who
don't prefix their objdirs with the string 'obj'. This attempts to ignore all
objdirs by looking for 'config.status' files in the topsrcdir.
This wasn't a huge issue before because BuildReader._relevant_mozbuild_finder
was only used in a handful of obscure edge cases. But now it's being invoked by
the taskgraph and consequently |mach try|. So we'll see this issue pop up for
people more frequently.
Differential Revision: https://phabricator.services.mozilla.com/D57113
--HG--
extra : moz-landing-system : lando
This change modifies all tests that use key3/cert8 to use the new files. It
removes test_sdr_upgraded_with_password, as without the upgrade part that is now
the same test as test_sdr_preexisting_with_password.
Differential Revision: https://phabricator.services.mozilla.com/D55708
--HG--
rename : security/manager/ssl/tests/unit/test_sdr_preexisting/key4.db => security/manager/ssl/tests/unit/test_broken_fips/key4.db
extra : moz-landing-system : lando
This change modifies all tests that use key3/cert8 to use the new files. It
removes test_sdr_upgraded_with_password, as without the upgrade part that is now
the same test as test_sdr_preexisting_with_password.
Differential Revision: https://phabricator.services.mozilla.com/D55708
--HG--
rename : security/manager/ssl/tests/unit/test_sdr_preexisting/key4.db => security/manager/ssl/tests/unit/test_broken_fips/key4.db
extra : moz-landing-system : lando
This change removes the legacy libnssdbm database that we migrated away from since Firefox 60.
This change modifies all tests that use key3/cert8 to use the new files. It
removes test_sdr_upgraded_with_password, as without the upgrade part that is now
the same test as test_sdr_preexisting_with_password. It otherwise removes support for libnssdbm everywhere in Gecko.
Differential Revision: https://phabricator.services.mozilla.com/D55708
--HG--
rename : security/manager/ssl/tests/unit/test_sdr_preexisting/key4.db => security/manager/ssl/tests/unit/test_broken_fips/key4.db
extra : moz-landing-system : lando
This optionally uses the rust-based `dump_syms` toolchain if it has been
installed, otherwise it uses the locally built version.
Differential Revision: https://phabricator.services.mozilla.com/D53320
--HG--
extra : moz-landing-system : lando
Bug 1295542 added empty chrome.manifest files to prevent malware from abusing
them. This workaround is no longer necessary because Firefox stopped reading
chrome.manifest outside omni.ja since bug 1543761.
Differential Revision: https://phabricator.services.mozilla.com/D55954
--HG--
extra : moz-landing-system : lando
Make the $PYTHON3 build var point to a full virtualenv bootstrapped with
the same libraries as the $PYTHON Python 2 build var. This allows us to
upgrade build tasks from $PYTHON to $PYTHON3.
This patch adds some debug logging and documentation to the Python
2 virtualenv so that it is easier to diagnose issues that may arise
from running two different Python interpreters in re-entrant
multiprocess routines.
Differential Revision: https://phabricator.services.mozilla.com/D50819
--HG--
extra : moz-landing-system : lando
This optionally uses the rust-based `dump_syms` toolchain if it has been
installed, otherwise it uses the locally built version.
Differential Revision: https://phabricator.services.mozilla.com/D53320
--HG--
extra : moz-landing-system : lando
In 1-tier PGO builds that shared the objdir between the instrumented and
profile-use builds, the instrumented build objects used a different
suffix (.i_o) to separate them from the profile-use build (which uses
the default .o suffix). These builds are now always in separate objdirs,
and don't need special suffix rules anymore.
As a bonus, this helps fix an issue with buildid.cpp continually
rebuilding because libxul_so.list always lists the inputs as *.o, which
don't exist if we're using a .i_o suffix. Make would always re-create
buildid.cpp and therefore libxul.so in the instrumented build even when
nothing has changed.
Differential Revision: https://phabricator.services.mozilla.com/D56115
--HG--
extra : moz-landing-system : lando
Allow running the jsshell using ./mach run. This copies the debugger
parameter handling from desktop configurations.
Differential Revision: https://phabricator.services.mozilla.com/D56296
--HG--
extra : moz-landing-system : lando
Local PGO builds now use 3-tier machinery under the hood. Instead of a
single object directory that gets cleaned in between the instrumented
and profile-use builds, now the instrumented build uses a separate
'${objdir}/instrumented' directory. This makes it easier to handle
within mach since we can drive the two builds with environment variables
and keep all build artifacts separate, without needing to do manual
cleanup in between.
Differential Revision: https://phabricator.services.mozilla.com/D50098
--HG--
extra : moz-landing-system : lando
This patch addresses the following issues which prevent building the
documentation on a Windows system:
1. check_jsdoc attempts to launch a file by the name of jsdoc using
subprocess.check_output(). But, the file jsdoc is a Bash script,
so it can't be launched this way on Windows. We should find the
correct file name to use instead (likely jsdoc.cmd).
2. The Windows CopyFile API will fail if the destination directory does
not exist. The procedure that creates the staging directory for Sphinx
attempts to use this API before any directories are created, so most calls
to it fail and most files don't get staged. These errors are ignored, so
the result is that Sphinx succeeds in generating a set of documentation
that is mostly empty.
3. Several instances where manually constructed paths that contain both back
and forward slashes are compared to or otherwise used with normalized paths,
resulting in incorrect behaviors.
Differential Revision: https://phabricator.services.mozilla.com/D54549
--HG--
extra : moz-landing-system : lando
Some of these were obvious typos. Others probably reflect once-correct components
that have been combined, split, or otherwise obsoleted; for these I've tried to
use the component associated with the bugs for recent changes to the affected files.
Differential Revision: https://phabricator.services.mozilla.com/D55756
--HG--
extra : moz-landing-system : lando
Remove the "clever" logic from patch D53070, which was meant to reduce code duplication, and replace with easier-to-understand logic. Copy-and-paste the previous version of the CompileFlags class before patch D53070 to ensure that we stamp out the regression.
Differential Revision: https://phabricator.services.mozilla.com/D55472
--HG--
extra : moz-landing-system : lando
This change does not fully enable C++17, as we still need standard
library support from some platforms.
Differential Revision: https://phabricator.services.mozilla.com/D54202
--HG--
extra : moz-landing-system : lando
If Mercurial (hg) is not installed on a Centos/Fedora system, execution
of:
$ ./mach bootstrap
will be failed as the bootstrap script tries to execute:
# dnf update mercurial
This commit adds 'dnf install' command if the mercurial wasn't
installed on the system. This command will try to execute installation
of mercurial.
In other case there will be a try to upgrade to the latest possibly version
of mercurial.
Differential Revision: https://phabricator.services.mozilla.com/D55639
--HG--
extra : moz-landing-system : lando
This change does not fully enable C++17, as we still need standard
library support from some platforms.
Differential Revision: https://phabricator.services.mozilla.com/D54202
--HG--
extra : moz-landing-system : lando
Add a proper title and popup attributes to page-options button.
Make recommended card's add-on names headings.
Give the HTML pane a title so it reads better in screen readers.
Always include a label for the search box.
Clarify the label on the extension enable checkbox.
Differential Revision: https://phabricator.services.mozilla.com/D54874
--HG--
extra : moz-landing-system : lando
Ensure that environment variables with unicode values are handled correctly
when setting up a virtualenv.
Differential Revision: https://phabricator.services.mozilla.com/D55019
--HG--
extra : moz-landing-system : lando
Add backend stuff to build sandboxed wasm libraries. (Don't actually update any moz.build files to consume this yet.)
Differential Revision: https://phabricator.services.mozilla.com/D54152
--HG--
extra : moz-landing-system : lando
Make the $PYTHON3 build var point to a full virtualenv bootstrapped with
the same libraries as the $PYTHON Python 2 build var. This allows us to
upgrade build tasks from $PYTHON to $PYTHON3.
This patch adds some debug logging and documentation to the Python
2 virtualenv so that it is easier to diagnose issues that may arise
from running two different Python interpreters in re-entrant
multiprocess routines.
Differential Revision: https://phabricator.services.mozilla.com/D50819
--HG--
extra : moz-landing-system : lando
Homebrew on OS X will change Python's sys.executable to a custom value
which messes with mach's virtualenv handling code. Override Homebrew's
changes with the correct sys.executable value.
Differential Revision: https://phabricator.services.mozilla.com/D54602
--HG--
extra : moz-landing-system : lando
Changes:
Remove `environment` from the python3 blacklist.
Remove the option to specify `configure` for the formatting argument, since it has never been supported (the method has never been implemented).
Differential Revision: https://phabricator.services.mozilla.com/D54395
--HG--
extra : moz-landing-system : lando
The FileAvoidWrite class does a bunch of stuff with bytes and strings
that doesn't work in Python 3. Make sure the object is handling only
bytes or only strings under the hood so Python 3 is happy.
The FileAvoidWrite unit tests written with MockedOpen don't work in
Python 3 either. Swap them out for vanilla pytest tests without
the MockedOpen dependency.
Differential Revision: https://phabricator.services.mozilla.com/D51341
--HG--
extra : moz-landing-system : lando
This will install ipython into the default virtualenv if it doesn't exist. Unless --no-virtualenv
is specified in which case an error will be printed.
Differential Revision: https://phabricator.services.mozilla.com/D53030
--HG--
extra : moz-landing-system : lando
`os.path.abspath` on Windows doesn't return a consistent value -- the capitalization can vary based on environmental factors that I don't fully understand. Regardless, normalize case where this function is called in config_status.
Differential Revision: https://phabricator.services.mozilla.com/D53889
--HG--
extra : moz-landing-system : lando
If configure is invoked manually, clobber.py is bypassed and the CLOBBER
file doesn't get touched. The clobber check in Makefile.in gets
triggered causing the build to stop.
Moving the objdir/CLOBBER file creation into configure.py should cause
it to be created regardless of how configure is invoked.
Depends on D53290
Differential Revision: https://phabricator.services.mozilla.com/D53291
--HG--
extra : moz-landing-system : lando
Bug 1596868 was triggered by forgetting a comma, turning an intended
tuple into a string, which Python dutifully iterated over as a sequence
of single characters. Let's add some basic typechecking so these sort
of mistakes don't happen again.
Differential Revision: https://phabricator.services.mozilla.com/D53815
--HG--
extra : moz-landing-system : lando
Now since we use unicode_literals by bug 1210157, mercurial version check will be error due to `TypeError: environment can only contain strings`.
So we don't use unicode for os.environment['PATH'] when appending Java path.
Differential Revision: https://phabricator.services.mozilla.com/D53139
--HG--
extra : moz-landing-system : lando
There's a lot of formatting that is irrelevant for Fluent,
and it's hard to diff usefully on the text file.
Instead, normalize the Fluent files by parsing them, and serialzing
them with Junk included.
Differential Revision: https://phabricator.services.mozilla.com/D53202
--HG--
extra : moz-landing-system : lando
The minidump-analyzer tool was originally conceived to be used from the crash
report client and as such was installed in the crash reporter client
application bundle on macOS. It was later adapted to work from Firefox itself
but this caused linking problems when invoked from the Firefox app bundle.
This patch moves the minidump-analyzer into the Firefox app bundle and adapts
the relevant code to find it there.
The minidump-analyzer was also not signed like the rest of our executables and
this patch addresses that issue too.
Differential Revision: https://phabricator.services.mozilla.com/D52910
--HG--
extra : moz-landing-system : lando
We need this for "full" C++17 support (everything is supported, but some
C++17 features still have bugs) and this change also brings Linux into
parity with our Mac requirements.
MANUAL PUSH: build toolchains on inbound to avoid clogging autoland
Differential Revision: https://phabricator.services.mozilla.com/D51450
This loader uses 'reader.find_variables_from_ast' to parse all *_MANIFESTS variables from
moz.build files using the abstract syntax tree. This means it will find all such variables
regardless of the current buildconfig.
Differential Revision: https://phabricator.services.mozilla.com/D51834
--HG--
extra : moz-landing-system : lando
Despite what the comment says, the finder *does* pick up the root moz.build. So
we end up processing it twice. This was never caught before because we only ever
used this function to read Sphinx related variables, of which the root moz.build
contains none.
Differential Revision: https://phabricator.services.mozilla.com/D51832
--HG--
extra : moz-landing-system : lando
This also moves the 'mach' docs from /python/mach to /mach. The reason being
that 'mach' doesn't really have anything to do with Python other than its
implemented in it.
Differential Revision: https://phabricator.services.mozilla.com/D52253
--HG--
extra : moz-landing-system : lando
Changes:
- use regular strings instead of byte strings when matching
- encode files in UTF-8 prior to hashing
Depends on D51414
Differential Revision: https://phabricator.services.mozilla.com/D51728
--HG--
extra : moz-landing-system : lando
Changes:
- change `urlparse` import to `urllib.parse` to make switch over to python3
Depends on D51414
Differential Revision: https://phabricator.services.mozilla.com/D51648
--HG--
extra : moz-landing-system : lando
Changes:
- update sections of `generate_sources_mozbuild.py` and `cmakeparser.py` to be python3 compatible
- change import of `urlparse` to be python3 compatible
Depends on D51414
Differential Revision: https://phabricator.services.mozilla.com/D51692
--HG--
extra : moz-landing-system : lando
The help view copies strings from the main menubar. When we moved the original DTD string
to ftl, there were performance implications for using it in browser.xhtml, so it was
only added once needed. The help view copies attributes from the items in the main menubar's
help menu, and so didn't copy the label for this item, resulting in the broken
behaviour.
To fix this, it's enough to have the string in the markup. As we've moved the other strings
into menubar.ftl, I'm taking the opportunity to move this string there, too, next to its
sibling string to report deceptive sites.
Differential Revision: https://phabricator.services.mozilla.com/D51850
--HG--
extra : moz-landing-system : lando
Add test annotations to the ReftestManifest and TestResolver. For example, a
test description from the TestResolver might now contain 'skip-if': 'skiaContent';
similar to the content provided for manifestparser tests; this will allow
'mach test-info report' to filter tests based on reftest manifest test
annotations.
Also add the referenced-test field which identifies the test file associated
with test entries for reference files; this will allow test-verify to
run the correct reftest when only the reference file has been modified.
Differential Revision: https://phabricator.services.mozilla.com/D51113
--HG--
extra : moz-landing-system : lando
This allows subclasses of MozbuildObject to define additional instance
arguments and still use 'from_environment'.
Differential Revision: https://phabricator.services.mozilla.com/D51174
--HG--
extra : moz-landing-system : lando
On Windows, './mach bootstrap' downloads 8 artifacts taking up 1GiB+ of
space. Given the current limits of 6 artifacts and 1GiB, this means a
full run of './mach bootstrap' doesn't actually use anything from the
cache. Each new artifact is over the limits, so the oldest artifact gets
removed from the cache before downloading the next one. Doubling the
limits here should give us some space to work with.
Depends on D50676
Differential Revision: https://phabricator.services.mozilla.com/D50677
--HG--
extra : moz-landing-system : lando
Artifacts are fetched to a local cache, and if the artifact in the cache
exists it won't be re-downloaded. However, the messaging suggested that
artifacts were always being downloaded when they were just re-used from
the cache, leading to confusion.
Differential Revision: https://phabricator.services.mozilla.com/D50676
--HG--
extra : moz-landing-system : lando
Separating these changes out into a separate patch/bug makes them a
little easier to verify. Turning on C++17 support should then be just a
matter of updating expected results.
Differential Revision: https://phabricator.services.mozilla.com/D47324
--HG--
extra : moz-landing-system : lando
Changes:
- remove the `unicode_literals` import from `test_archive.py` as it was causing failures when dealing with file paths
- mark failing tests with `xfail` annotations, to be investigated at a later date
Differential Revision: https://phabricator.services.mozilla.com/D49498
--HG--
extra : moz-landing-system : lando
Since bootstrap now pulls mozilla-unified it makes sense to reference the
unified repo as well as central.
Differential Revision: https://phabricator.services.mozilla.com/D49478
--HG--
extra : moz-landing-system : lando
This is needed to release a new mozlog with the PRECONDITION_FAILED
test and subtest status for use in web-platform-tests.
Update all in-tree dependencies on mozlog to >=5.0. These were found
with `hg grep 'mozlog.*[0-9]'`.
Only testing/web-platform/tests/tools/wptrunner/requirements.txt
remains on 4.2.0, and it will be updated in upstream wpt after mozlog
5.0 has been released.
Differential Revision: https://phabricator.services.mozilla.com/D50456
--HG--
extra : moz-landing-system : lando
Add code to clean up Git and Hg repositories, and invoke that where `./mach vendor rust` would throw an error. Unfortunately, `cargo vendor` also updates the repo's root `Cargo.lock` file in-place and while we could `git checkout`/`hg revert` that file for the user, `Cargo.lock` may have had pre-existing changes that would be overwritten by such a change. Instead of a potentially destructive update in the error case, I've opted to add an extra error message to tell you how to deal with that one file.
Differential Revision: https://phabricator.services.mozilla.com/D49494
--HG--
extra : moz-landing-system : lando
Changes:
- remove `uuid` from the python3 blacklist in `mach`
- enable `test_telemetry.py` for python3
- adjust test outcome expectation for Windows + python3, suspect `mozpack.path` is not filtering Windows path correctly in python3 environment
- switch file read mode between `r` and `rb` depending on version of python
Differential Revision: https://phabricator.services.mozilla.com/D45903
--HG--
extra : moz-landing-system : lando
The BuildReader's 'find_sphinx_variables' function is hardcoded to look for the
SPHINX_TREES and SPHINX_PYTHON_PACKAGE_DIRS variables. But it's otherwise
implemented to find any arbitrary variable (as long as they are a simple list
or dict).
This change simply moves the variable name(s) to the function call. The comment
about possibly wanted to use a higher level AST library to parse other kinds of
variables still applies, but for now this change is good enough to suit my
needs.
Differential Revision: https://phabricator.services.mozilla.com/D48424
--HG--
extra : moz-landing-system : lando
Fennec is removed, but mach install still tries to install Fennec. It should
install GVE instead.
Differential Revision: https://phabricator.services.mozilla.com/D48811
--HG--
extra : moz-landing-system : lando
Some generated files are missing in generated-sources.json so just add them.
Differential Revision: https://phabricator.services.mozilla.com/D48085
--HG--
extra : moz-landing-system : lando
This leaves the testharness files, because they are used in various mochitests.
Differential Revision: https://phabricator.services.mozilla.com/D49132
--HG--
extra : moz-landing-system : lando
This leaves the testharness files, because they are used in mochitest-chrome
tests in dom/animation/test.
Differential Revision: https://phabricator.services.mozilla.com/D49132
--HG--
extra : moz-landing-system : lando
This leaves the testharness files, because they are used in mochitest-chrome
tests in dom/animation/test.
Differential Revision: https://phabricator.services.mozilla.com/D49132
--HG--
extra : moz-landing-system : lando
Some fields of /etc/os-release are optional, so do not throw KeyError's when they're missing.
Differential Revision: https://phabricator.services.mozilla.com/D48984
--HG--
extra : moz-landing-system : lando
Adds a way for mochitest, reftest, and crashtests to skip XBL related
tests when XBL is disabled. Also, add an app constant so JS can
check whether XBL is enabled.
Depends on D45614
Differential Revision: https://phabricator.services.mozilla.com/D45615
--HG--
extra : moz-landing-system : lando
This does many things:
1) stops producing (and consuming) `FennecJNI*` JNI wrappers
2) removes the :app and :thirdparty Gradle projects
3) removes relevant pieces of the Gradle target configuration
4) updates lints
5) purges old configurations
After this commit, the `mobile/android` project/application builds
only GeckoView.
Differential Revision: https://phabricator.services.mozilla.com/D46536
--HG--
extra : moz-landing-system : lando
Environment should be bytes on Python 2 (to avoid Windows errors) and text on
Python 3. The 'ensure_subprocess' env utility function handles this.
Differential Revision: https://phabricator.services.mozilla.com/D48115
--HG--
extra : moz-landing-system : lando
This logic is very 'mozill-central' specific and should live outside of mach
core if possible. Luckily we already have a concept of a 'pre_dispatch_handler'
that is meant for exactly this type of use case.
Differential Revision: https://phabricator.services.mozilla.com/D47668
--HG--
extra : moz-landing-system : lando
Nathan Froyd
Razvan Maries
Jonathan Watt
Andrew Halberstadt
Coroiu Cristina
Dan Mosedale
Ricky Stewart
Bogdan Tara
Mike Shal
Zibi Braniecki
Sylvestre Ledru
Chris Manchester
Oana Pop Rus
J.C. Jones
Csoregi Natalia
Andreea Pavel
Eric Rahm
Noemi Erli
Masatoshi Kimura
Māris Fogels
Dorel Luca
Ted Campbell
David Major
Molly Howell
Florin Strugariu
Geoff Brown
Gurzau Raul
Sebastian Hengst
shindli
Alexander Kuleshov
Mihai Alexandru Michis
Andi-Bogdan Postelnicu
Francesco Lodolo (:flod)
Mark Striemer
Botond Ballo
Edwin Takahashi
Cosmin Sabou
Geoff Lankow
Makoto Kato
Alexander Surkov
Axel Hecht
Tom Schuster
Gabriele Svelto
Jesse Schwartzentruber
Nick Thomas
Kirk Steuber
Tom Prince
Gijs Kruitbosch
Bryce Seager van Dyk
Philip Jägenstedt
Anthony Ramine
Marco Castelluccio
Wes Kocher
Calixte Denizet
Edouard Oger
Ms2ger
Kenny Levinsen
Brendan Dahl
Mark Banner
Bastien Abadie
Nick Alexander
Anmol Agarwal
Narcis Beleuzu
Jared Wein
Brian Grinstead