This gets rid of the last use of the 'which' module left in the tree. So not
only will this help 'configure' become a little more Python 3 compatible, but
we can now remove 'third_party/python/which'.
Differential Revision: https://phabricator.services.mozilla.com/D37427
--HG--
extra : moz-landing-system : lando
This gets rid of the last use of the 'which' module left in the tree. So not
only will this help 'configure' become a little more Python 3 compatible, but
we can now remove 'third_party/python/which'.
Differential Revision: https://phabricator.services.mozilla.com/D37427
--HG--
extra : moz-landing-system : lando
Since we have a limit of the number of arguments length that the shell can support we have to
pass the files for the analysis to `run-clang-tidy.py` in chuncks, calculated in proportion
to the number or logical cpu cores.
Differential Revision: https://phabricator.services.mozilla.com/D58971
--HG--
extra : moz-landing-system : lando
Many of the utility classes very heavily rely on multiple inheritance which is broadly unncessary and made debugging difficult, so I ripped it out wherever it was necessary. The indented_repr() function is also more or less a re-implementation of pprint, so to support Python 3 I've instead made it a small wrapper around pprint. (This results in the output of the function being slightly different than we're used to, as pprint does sorting/indentation/line breaking differently than indented_repr() currently does, but it'll be nice to not have to maintain this code that is slightly different than pprint for no real reason.)
Differential Revision: https://phabricator.services.mozilla.com/D58306
--HG--
extra : moz-landing-system : lando
Make the choice of VCS work via an interactive prompt rather than by
command line argument, just like we do for all other choices.
Differential Revision: https://phabricator.services.mozilla.com/D58614
--HG--
extra : moz-landing-system : lando
Warning flags might have been generated for a different compiler, so we
can't use them here. The same argument applies to coverage flags.
Depends on D58070
Differential Revision: https://phabricator.services.mozilla.com/D58071
--HG--
extra : moz-landing-system : lando
The optimization flags that may have been chosen for our target compiler
are not necessarily appropriate for our wasm compiler.
Differential Revision: https://phabricator.services.mozilla.com/D58070
--HG--
extra : moz-landing-system : lando
There was only a single use of 'install-to-subdir' in all our manifests (which
was removed in the previous commit). All the comments around it suggest that it
was a giant hack. Removing it cleans up a lot of complexity and saves me the
trouble of needing to support it in the new sourcedir based TestResolver.
Differential Revision: https://phabricator.services.mozilla.com/D57170
--HG--
extra : moz-landing-system : lando
By default we create both the RecursiveMake and FasterMake backends, but
building.py is only checking if the RecursiveMake backend is out of date
to decide if the backend needs to be updated. Since the FasterMake
backend may have different input files, we also need to check if that's
out of date, otherwise renaming certain files can cause './mach build
faster' to break.
Differential Revision: https://phabricator.services.mozilla.com/D58010
--HG--
extra : moz-landing-system : lando
Change import order.
Enable python3 testing for `test_copier.py`.
In the test methods, ensure the returned file names are decoded as UTF-8 by default.
In the test methods, check for byte-string by default; in python2, this would be treated like the standard string.
Differential Revision: https://phabricator.services.mozilla.com/D55187
--HG--
extra : moz-landing-system : lando
This undoes the mistaken move of this directory into third_party that happened
as part of bug 1346025.
The .flake8 changes are required because lldbutils used to be excluded from
linting by virtue of being inside third_party, but would otherwise be subject
to it after this move. Fixing the existing lint errors will happen in:
- https://bugzilla.mozilla.org/show_bug.cgi?id=1605144
- https://bugzilla.mozilla.org/show_bug.cgi?id=1605145
Differential Revision: https://phabricator.services.mozilla.com/D57141
--HG--
rename : third_party/python/lldbutils/README.txt => python/lldbutils/README.txt
rename : third_party/python/lldbutils/lldbutils/__init__.py => python/lldbutils/lldbutils/__init__.py
rename : third_party/python/lldbutils/lldbutils/content.py => python/lldbutils/lldbutils/content.py
rename : third_party/python/lldbutils/lldbutils/general.py => python/lldbutils/lldbutils/general.py
rename : third_party/python/lldbutils/lldbutils/gfx.py => python/lldbutils/lldbutils/gfx.py
rename : third_party/python/lldbutils/lldbutils/layout.py => python/lldbutils/lldbutils/layout.py
rename : third_party/python/lldbutils/lldbutils/utils.py => python/lldbutils/lldbutils/utils.py
extra : moz-landing-system : lando
We recently updated the version of pip we use, and it no longer supports
--use-wheels.
Differential Revision: https://phabricator.services.mozilla.com/D57404
--HG--
extra : moz-landing-system : lando
This patch causes the newest version of NodeJS 8.17.0 to be used to build the mozilla toolchains installed on the CI machines as well as by `mach bootstrap`, which in turn gets an important security fix by updating to npm 6.13.4.
This patch also causes mach commands which call npm to abort if the version in use is older than 6.13.4 to avoid (especially) exploits on local developers that exfiltrate their data.
Differential Revision: https://phabricator.services.mozilla.com/D57674
--HG--
extra : moz-landing-system : lando
If mercurial is already installed via pip (or possibly other means),
then mach bootstrap will fail. We should check if we already have a
sufficient mercurial before forcing it to be installed with
ports/homebrew.
Differential Revision: https://phabricator.services.mozilla.com/D57566
--HG--
extra : moz-landing-system : lando
Currently BuildReader._relevant_mozbuild_finder attempted to exclude objdirs by
ignoring the pattern 'obj*'. However this can cause problems for developers who
don't prefix their objdirs with the string 'obj'. This attempts to ignore all
objdirs by looking for 'config.status' files in the topsrcdir.
This wasn't a huge issue before because BuildReader._relevant_mozbuild_finder
was only used in a handful of obscure edge cases. But now it's being invoked by
the taskgraph and consequently |mach try|. So we'll see this issue pop up for
people more frequently.
Differential Revision: https://phabricator.services.mozilla.com/D57113
--HG--
extra : moz-landing-system : lando
This change modifies all tests that use key3/cert8 to use the new files. It
removes test_sdr_upgraded_with_password, as without the upgrade part that is now
the same test as test_sdr_preexisting_with_password.
Differential Revision: https://phabricator.services.mozilla.com/D55708
--HG--
rename : security/manager/ssl/tests/unit/test_sdr_preexisting/key4.db => security/manager/ssl/tests/unit/test_broken_fips/key4.db
extra : moz-landing-system : lando
This change modifies all tests that use key3/cert8 to use the new files. It
removes test_sdr_upgraded_with_password, as without the upgrade part that is now
the same test as test_sdr_preexisting_with_password.
Differential Revision: https://phabricator.services.mozilla.com/D55708
--HG--
rename : security/manager/ssl/tests/unit/test_sdr_preexisting/key4.db => security/manager/ssl/tests/unit/test_broken_fips/key4.db
extra : moz-landing-system : lando
This change removes the legacy libnssdbm database that we migrated away from since Firefox 60.
This change modifies all tests that use key3/cert8 to use the new files. It
removes test_sdr_upgraded_with_password, as without the upgrade part that is now
the same test as test_sdr_preexisting_with_password. It otherwise removes support for libnssdbm everywhere in Gecko.
Differential Revision: https://phabricator.services.mozilla.com/D55708
--HG--
rename : security/manager/ssl/tests/unit/test_sdr_preexisting/key4.db => security/manager/ssl/tests/unit/test_broken_fips/key4.db
extra : moz-landing-system : lando
This optionally uses the rust-based `dump_syms` toolchain if it has been
installed, otherwise it uses the locally built version.
Differential Revision: https://phabricator.services.mozilla.com/D53320
--HG--
extra : moz-landing-system : lando
Bug 1295542 added empty chrome.manifest files to prevent malware from abusing
them. This workaround is no longer necessary because Firefox stopped reading
chrome.manifest outside omni.ja since bug 1543761.
Differential Revision: https://phabricator.services.mozilla.com/D55954
--HG--
extra : moz-landing-system : lando
Make the $PYTHON3 build var point to a full virtualenv bootstrapped with
the same libraries as the $PYTHON Python 2 build var. This allows us to
upgrade build tasks from $PYTHON to $PYTHON3.
This patch adds some debug logging and documentation to the Python
2 virtualenv so that it is easier to diagnose issues that may arise
from running two different Python interpreters in re-entrant
multiprocess routines.
Differential Revision: https://phabricator.services.mozilla.com/D50819
--HG--
extra : moz-landing-system : lando
This optionally uses the rust-based `dump_syms` toolchain if it has been
installed, otherwise it uses the locally built version.
Differential Revision: https://phabricator.services.mozilla.com/D53320
--HG--
extra : moz-landing-system : lando
In 1-tier PGO builds that shared the objdir between the instrumented and
profile-use builds, the instrumented build objects used a different
suffix (.i_o) to separate them from the profile-use build (which uses
the default .o suffix). These builds are now always in separate objdirs,
and don't need special suffix rules anymore.
As a bonus, this helps fix an issue with buildid.cpp continually
rebuilding because libxul_so.list always lists the inputs as *.o, which
don't exist if we're using a .i_o suffix. Make would always re-create
buildid.cpp and therefore libxul.so in the instrumented build even when
nothing has changed.
Differential Revision: https://phabricator.services.mozilla.com/D56115
--HG--
extra : moz-landing-system : lando
Allow running the jsshell using ./mach run. This copies the debugger
parameter handling from desktop configurations.
Differential Revision: https://phabricator.services.mozilla.com/D56296
--HG--
extra : moz-landing-system : lando
Local PGO builds now use 3-tier machinery under the hood. Instead of a
single object directory that gets cleaned in between the instrumented
and profile-use builds, now the instrumented build uses a separate
'${objdir}/instrumented' directory. This makes it easier to handle
within mach since we can drive the two builds with environment variables
and keep all build artifacts separate, without needing to do manual
cleanup in between.
Differential Revision: https://phabricator.services.mozilla.com/D50098
--HG--
extra : moz-landing-system : lando
This patch addresses the following issues which prevent building the
documentation on a Windows system:
1. check_jsdoc attempts to launch a file by the name of jsdoc using
subprocess.check_output(). But, the file jsdoc is a Bash script,
so it can't be launched this way on Windows. We should find the
correct file name to use instead (likely jsdoc.cmd).
2. The Windows CopyFile API will fail if the destination directory does
not exist. The procedure that creates the staging directory for Sphinx
attempts to use this API before any directories are created, so most calls
to it fail and most files don't get staged. These errors are ignored, so
the result is that Sphinx succeeds in generating a set of documentation
that is mostly empty.
3. Several instances where manually constructed paths that contain both back
and forward slashes are compared to or otherwise used with normalized paths,
resulting in incorrect behaviors.
Differential Revision: https://phabricator.services.mozilla.com/D54549
--HG--
extra : moz-landing-system : lando
Some of these were obvious typos. Others probably reflect once-correct components
that have been combined, split, or otherwise obsoleted; for these I've tried to
use the component associated with the bugs for recent changes to the affected files.
Differential Revision: https://phabricator.services.mozilla.com/D55756
--HG--
extra : moz-landing-system : lando
Remove the "clever" logic from patch D53070, which was meant to reduce code duplication, and replace with easier-to-understand logic. Copy-and-paste the previous version of the CompileFlags class before patch D53070 to ensure that we stamp out the regression.
Differential Revision: https://phabricator.services.mozilla.com/D55472
--HG--
extra : moz-landing-system : lando
This change does not fully enable C++17, as we still need standard
library support from some platforms.
Differential Revision: https://phabricator.services.mozilla.com/D54202
--HG--
extra : moz-landing-system : lando
If Mercurial (hg) is not installed on a Centos/Fedora system, execution
of:
$ ./mach bootstrap
will be failed as the bootstrap script tries to execute:
# dnf update mercurial
This commit adds 'dnf install' command if the mercurial wasn't
installed on the system. This command will try to execute installation
of mercurial.
In other case there will be a try to upgrade to the latest possibly version
of mercurial.
Differential Revision: https://phabricator.services.mozilla.com/D55639
--HG--
extra : moz-landing-system : lando
This change does not fully enable C++17, as we still need standard
library support from some platforms.
Differential Revision: https://phabricator.services.mozilla.com/D54202
--HG--
extra : moz-landing-system : lando
Add a proper title and popup attributes to page-options button.
Make recommended card's add-on names headings.
Give the HTML pane a title so it reads better in screen readers.
Always include a label for the search box.
Clarify the label on the extension enable checkbox.
Differential Revision: https://phabricator.services.mozilla.com/D54874
--HG--
extra : moz-landing-system : lando
Ensure that environment variables with unicode values are handled correctly
when setting up a virtualenv.
Differential Revision: https://phabricator.services.mozilla.com/D55019
--HG--
extra : moz-landing-system : lando
Add backend stuff to build sandboxed wasm libraries. (Don't actually update any moz.build files to consume this yet.)
Differential Revision: https://phabricator.services.mozilla.com/D54152
--HG--
extra : moz-landing-system : lando
Make the $PYTHON3 build var point to a full virtualenv bootstrapped with
the same libraries as the $PYTHON Python 2 build var. This allows us to
upgrade build tasks from $PYTHON to $PYTHON3.
This patch adds some debug logging and documentation to the Python
2 virtualenv so that it is easier to diagnose issues that may arise
from running two different Python interpreters in re-entrant
multiprocess routines.
Differential Revision: https://phabricator.services.mozilla.com/D50819
--HG--
extra : moz-landing-system : lando
Homebrew on OS X will change Python's sys.executable to a custom value
which messes with mach's virtualenv handling code. Override Homebrew's
changes with the correct sys.executable value.
Differential Revision: https://phabricator.services.mozilla.com/D54602
--HG--
extra : moz-landing-system : lando
Changes:
Remove `environment` from the python3 blacklist.
Remove the option to specify `configure` for the formatting argument, since it has never been supported (the method has never been implemented).
Differential Revision: https://phabricator.services.mozilla.com/D54395
--HG--
extra : moz-landing-system : lando
The FileAvoidWrite class does a bunch of stuff with bytes and strings
that doesn't work in Python 3. Make sure the object is handling only
bytes or only strings under the hood so Python 3 is happy.
The FileAvoidWrite unit tests written with MockedOpen don't work in
Python 3 either. Swap them out for vanilla pytest tests without
the MockedOpen dependency.
Differential Revision: https://phabricator.services.mozilla.com/D51341
--HG--
extra : moz-landing-system : lando
This will install ipython into the default virtualenv if it doesn't exist. Unless --no-virtualenv
is specified in which case an error will be printed.
Differential Revision: https://phabricator.services.mozilla.com/D53030
--HG--
extra : moz-landing-system : lando
`os.path.abspath` on Windows doesn't return a consistent value -- the capitalization can vary based on environmental factors that I don't fully understand. Regardless, normalize case where this function is called in config_status.
Differential Revision: https://phabricator.services.mozilla.com/D53889
--HG--
extra : moz-landing-system : lando
If configure is invoked manually, clobber.py is bypassed and the CLOBBER
file doesn't get touched. The clobber check in Makefile.in gets
triggered causing the build to stop.
Moving the objdir/CLOBBER file creation into configure.py should cause
it to be created regardless of how configure is invoked.
Depends on D53290
Differential Revision: https://phabricator.services.mozilla.com/D53291
--HG--
extra : moz-landing-system : lando
Bug 1596868 was triggered by forgetting a comma, turning an intended
tuple into a string, which Python dutifully iterated over as a sequence
of single characters. Let's add some basic typechecking so these sort
of mistakes don't happen again.
Differential Revision: https://phabricator.services.mozilla.com/D53815
--HG--
extra : moz-landing-system : lando
Now since we use unicode_literals by bug 1210157, mercurial version check will be error due to `TypeError: environment can only contain strings`.
So we don't use unicode for os.environment['PATH'] when appending Java path.
Differential Revision: https://phabricator.services.mozilla.com/D53139
--HG--
extra : moz-landing-system : lando
There's a lot of formatting that is irrelevant for Fluent,
and it's hard to diff usefully on the text file.
Instead, normalize the Fluent files by parsing them, and serialzing
them with Junk included.
Differential Revision: https://phabricator.services.mozilla.com/D53202
--HG--
extra : moz-landing-system : lando
The minidump-analyzer tool was originally conceived to be used from the crash
report client and as such was installed in the crash reporter client
application bundle on macOS. It was later adapted to work from Firefox itself
but this caused linking problems when invoked from the Firefox app bundle.
This patch moves the minidump-analyzer into the Firefox app bundle and adapts
the relevant code to find it there.
The minidump-analyzer was also not signed like the rest of our executables and
this patch addresses that issue too.
Differential Revision: https://phabricator.services.mozilla.com/D52910
--HG--
extra : moz-landing-system : lando
We need this for "full" C++17 support (everything is supported, but some
C++17 features still have bugs) and this change also brings Linux into
parity with our Mac requirements.
MANUAL PUSH: build toolchains on inbound to avoid clogging autoland
Differential Revision: https://phabricator.services.mozilla.com/D51450
This loader uses 'reader.find_variables_from_ast' to parse all *_MANIFESTS variables from
moz.build files using the abstract syntax tree. This means it will find all such variables
regardless of the current buildconfig.
Differential Revision: https://phabricator.services.mozilla.com/D51834
--HG--
extra : moz-landing-system : lando
Despite what the comment says, the finder *does* pick up the root moz.build. So
we end up processing it twice. This was never caught before because we only ever
used this function to read Sphinx related variables, of which the root moz.build
contains none.
Differential Revision: https://phabricator.services.mozilla.com/D51832
--HG--
extra : moz-landing-system : lando
This also moves the 'mach' docs from /python/mach to /mach. The reason being
that 'mach' doesn't really have anything to do with Python other than its
implemented in it.
Differential Revision: https://phabricator.services.mozilla.com/D52253
--HG--
extra : moz-landing-system : lando
Changes:
- use regular strings instead of byte strings when matching
- encode files in UTF-8 prior to hashing
Depends on D51414
Differential Revision: https://phabricator.services.mozilla.com/D51728
--HG--
extra : moz-landing-system : lando
Changes:
- change `urlparse` import to `urllib.parse` to make switch over to python3
Depends on D51414
Differential Revision: https://phabricator.services.mozilla.com/D51648
--HG--
extra : moz-landing-system : lando
Changes:
- update sections of `generate_sources_mozbuild.py` and `cmakeparser.py` to be python3 compatible
- change import of `urlparse` to be python3 compatible
Depends on D51414
Differential Revision: https://phabricator.services.mozilla.com/D51692
--HG--
extra : moz-landing-system : lando
The help view copies strings from the main menubar. When we moved the original DTD string
to ftl, there were performance implications for using it in browser.xhtml, so it was
only added once needed. The help view copies attributes from the items in the main menubar's
help menu, and so didn't copy the label for this item, resulting in the broken
behaviour.
To fix this, it's enough to have the string in the markup. As we've moved the other strings
into menubar.ftl, I'm taking the opportunity to move this string there, too, next to its
sibling string to report deceptive sites.
Differential Revision: https://phabricator.services.mozilla.com/D51850
--HG--
extra : moz-landing-system : lando
Add test annotations to the ReftestManifest and TestResolver. For example, a
test description from the TestResolver might now contain 'skip-if': 'skiaContent';
similar to the content provided for manifestparser tests; this will allow
'mach test-info report' to filter tests based on reftest manifest test
annotations.
Also add the referenced-test field which identifies the test file associated
with test entries for reference files; this will allow test-verify to
run the correct reftest when only the reference file has been modified.
Differential Revision: https://phabricator.services.mozilla.com/D51113
--HG--
extra : moz-landing-system : lando
This allows subclasses of MozbuildObject to define additional instance
arguments and still use 'from_environment'.
Differential Revision: https://phabricator.services.mozilla.com/D51174
--HG--
extra : moz-landing-system : lando
On Windows, './mach bootstrap' downloads 8 artifacts taking up 1GiB+ of
space. Given the current limits of 6 artifacts and 1GiB, this means a
full run of './mach bootstrap' doesn't actually use anything from the
cache. Each new artifact is over the limits, so the oldest artifact gets
removed from the cache before downloading the next one. Doubling the
limits here should give us some space to work with.
Depends on D50676
Differential Revision: https://phabricator.services.mozilla.com/D50677
--HG--
extra : moz-landing-system : lando
Artifacts are fetched to a local cache, and if the artifact in the cache
exists it won't be re-downloaded. However, the messaging suggested that
artifacts were always being downloaded when they were just re-used from
the cache, leading to confusion.
Differential Revision: https://phabricator.services.mozilla.com/D50676
--HG--
extra : moz-landing-system : lando
Separating these changes out into a separate patch/bug makes them a
little easier to verify. Turning on C++17 support should then be just a
matter of updating expected results.
Differential Revision: https://phabricator.services.mozilla.com/D47324
--HG--
extra : moz-landing-system : lando
Changes:
- remove the `unicode_literals` import from `test_archive.py` as it was causing failures when dealing with file paths
- mark failing tests with `xfail` annotations, to be investigated at a later date
Differential Revision: https://phabricator.services.mozilla.com/D49498
--HG--
extra : moz-landing-system : lando
Since bootstrap now pulls mozilla-unified it makes sense to reference the
unified repo as well as central.
Differential Revision: https://phabricator.services.mozilla.com/D49478
--HG--
extra : moz-landing-system : lando
This is needed to release a new mozlog with the PRECONDITION_FAILED
test and subtest status for use in web-platform-tests.
Update all in-tree dependencies on mozlog to >=5.0. These were found
with `hg grep 'mozlog.*[0-9]'`.
Only testing/web-platform/tests/tools/wptrunner/requirements.txt
remains on 4.2.0, and it will be updated in upstream wpt after mozlog
5.0 has been released.
Differential Revision: https://phabricator.services.mozilla.com/D50456
--HG--
extra : moz-landing-system : lando
Andrew Halberstadt
Coroiu Cristina
Ricky Stewart
Zibi Braniecki
Matt Singleton
James Graham
Andi-Bogdan Postelnicu
Francesco Lodolo (:flod)
Brian Grinstead
Sylvestre Ledru
Bogdan Tara
Ciure Andrei
Mihai Alexandru Michis
Geoff Lankow
Nathan Froyd
Mike Shal
Jonathan Watt
Edwin Takahashi
Razvan Maries
Dan Mosedale
Chris Manchester
Oana Pop Rus
J.C. Jones
Csoregi Natalia
Andreea Pavel
Eric Rahm
Noemi Erli
Masatoshi Kimura
Māris Fogels
Dorel Luca
Ted Campbell
David Major
Molly Howell
Florin Strugariu
Geoff Brown
Gurzau Raul
Sebastian Hengst
shindli
Alexander Kuleshov
Mark Striemer
Botond Ballo
Cosmin Sabou
Makoto Kato
Alexander Surkov
Axel Hecht
Tom Schuster
Gabriele Svelto
Jesse Schwartzentruber
Nick Thomas
Kirk Steuber
Tom Prince
Gijs Kruitbosch
Bryce Seager van Dyk
Philip Jägenstedt
Anthony Ramine
Marco Castelluccio