This also migrates the vcs.py test to mozversioncontrol and adds a new task for
it.
MozReview-Commit-ID: 9jTRkjNupVA
--HG--
extra : rebase_source : 400f27498e00ea45234ad7c951770b098e916b8e
Sometimes commands return non-zero even though everything is ok. For example,
'hg outgoing' returns 1 if there are no outgoing files. This adds a way for
specific function calls not to abort if something goes wrong. Instead, stderr
will be printed (if any) and an empty string is returned.
MozReview-Commit-ID: E089djeHrmr
--HG--
extra : rebase_source : be351f357705e09c3fe876cecefa7ddd9c48987e
This adds 'get_outgoing_files'. First it automatically attempts to find the
upstream remote the current change is based on, then returns all files changed
in the local branch.
If an upstream remote can't be detected, it raises MissingUpstreamRepo
MozReview-Commit-ID: 9zSB9EdwVU8
--HG--
extra : rebase_source : e352d6d471644ef9eda76b972b789d6b54449471
There's currently a function for getting added files (A) and modified files
(M). We'll also eventually need the ability to get deleted files (D) and any
combination of the above, e.g (AM). Rather than creating a new function for
each possible case, let's have a single function where you can pass in which
modifiers you are interested in. With this patch, if you want all modified and
added files, you can do:
get_changed_files('AM')
By default 'ADM' is used.
This also adds a 'mode' option for git. This allows consumers to return staged
files, unstaged files or both. The default ('unstaged') keeps the current
behaviour in tact.
MozReview-Commit-ID: 9IA1bxaJS80
--HG--
extra : rebase_source : 160f650220ca9a35b4b116bc9fa13f28d84419fa
Technically this turns on gnu++14. I encountered a few errors when using c++14:
1) _USE_MATH_DEFINES needed to be defined for MinGW
2) MinGW did not define _finite under c++14
3) MinGW's float.h did not define Microsoft specific float functions under c++14
All of these were because c++14 defines _STRICT_ANSI_ which MinGW obeys and
avoids defining certain functions. The first two could be patched around, but
the third was a blocker, so we switched to gnu++14
MozReview-Commit-ID: 6Y7gEQgApYp
--HG--
extra : rebase_source : dabbd40c049c36e780b585e0bef0a8e25887d089
Unfortunately this also needs to be kept in Makefile.in to handle
other consumers of INCLUDES while we transition them.
MozReview-Commit-ID: 9OYlu6Jv1XZ
--HG--
extra : rebase_source : 719200501a93e836a03a64b5e1cd950a8f2e696a
GCC isn't safe to use on architectures that switched to Clang because
libstdc++ and libc++ aren't very compatible. Newer LLVM and Clang are
often already installed as a dependency for Mesa packages. So, always
require llvm* package.
MozReview-Commit-ID: 8651mz5tiIp
--HG--
extra : rebase_source : 7713e167b34f14a18fd5bf9c5ec33e926b2b929c
Currently line linters (linters that open a file and process it line by line,
by applying a regex for example), don't handle directories. If a directory is
passed in, it will try to 'open' it, which fails. Directories can get hit if
the linter has a directory in its include directive or if the user passes in
--no-filter.
This patch modifies LineLinters so that if a directory is detected, we search
for all relevant files under that directory. If 'extensions' is used, we'll
look for only files with appropriate extensions. Otherwise we assume the
linter wants every file.
MozReview-Commit-ID: D9lzTNuQTob
--HG--
extra : rebase_source : 0b952c06eae28b67b687813ff7e75b231b2dd4d3
The MozillaBuildBootstrapper specific rust install code in not needed as
mozbase already includes genertic code to achieve the same outcome. The
mozilla-build specific code also leads to issues where it tries to add already
existing targets and fails the bootstrap. This changeset removes the
mozilla-build specific step.
MozReview-Commit-ID: G0BqKZrF40A
--HG--
extra : rebase_source : 60e9638afff744c937a9665d6fd5830187835ea4
The code for obtaining a BuildReader for evaluating moz.build files
is generic and non-trivial. We already had a custom implementation
for `mach file-info` that implemented support for Mercurial
integration. Bug 1383880 will introduce a second consumer.
So this commit factors out the "obtain a BuildReader" logic into
a reusable function on our base MozbuildObject class. This makes
it easily available to various parts of the build system and mach
commands.
As part of the change, we detect when ``.`` is being used as the
revision and verify the working directory is clean. This behavior
can be disabled via argument if unwanted. But it's useful by default
to ensure consumers aren't expecting to read uncommitted changes.
MozReview-Commit-ID: LeYFqAb3HAe
--HG--
extra : rebase_source : d5ed4e4f5570a58a68853188de2225cd4e64ab3a
This is generally useful functionality to have. A consume will be
introduced in an upcoming commit.
MozReview-Commit-ID: 4arTMfJSiEC
--HG--
extra : rebase_source : 4bcf70f58b57b79b8dcb7a6eed633e1c7e42aca3
It seems reasonable to expose this outside of the BuildReader.
MozReview-Commit-ID: 4paDbYl9dEd
--HG--
extra : rebase_source : 86bb559500952a40fc9afbf958be5706dd9f858e
This also passes the '--noreplace' option to all the emerge invocations thus
preventing already installed packages from being rebuilt from scratch.
MozReview-Commit-ID: 4JBuptmgS3Y
--HG--
extra : rebase_source : e581607d4a2e997e7d79c7c4496d13b8e9b10e50
To construct an empty set, we need to use the `set()` notation. In order
to do that, we need to expose `set` to the moz.build sandbox.
MozReview-Commit-ID: DMyKnF0FEx2
--HG--
extra : rebase_source : 5cfe8080ec333a1eca70cd3edba2aaaff6406820
And convert consumers to context managers because hglib requires that.
MozReview-Commit-ID: Ckf1yBYeUlm
--HG--
extra : rebase_source : 985220032bced1a7077fd9b04ca8ad6de822c887
Because hglib spawns a persistent process, we introduce a context
manager for Repository. It no-ops by default. On HgRepository it
controls the lifetime of the persistent hg process.
A helper method for running an hg command via hglib has been added.
We can't transition existing methods to hglib because hglib
requires a context manager, which no consumer is using yet.
MozReview-Commit-ID: 8z0fcGFeAm5
--HG--
extra : rebase_source : 1cbfe44a5aafe838c25bcde91cc1deaf5e7367de
This ensures we use forward slashes, even if Mercurial emits
backslashes (which it can do on Windows).
MozReview-Commit-ID: 2dnWAEvytwn
--HG--
extra : rebase_source : 9ebb454bc9ad11b3eba334e412685e529573a0a1
These tests weren't running in automation because hglib wasn't
available. An upcoming commit will vendor hglib. This exposed that the
tests can fail if ui.username isn't set.
In this commit, we introduce a helper function to obtain an hglib
client with ui.username set. We also convert tests to use the context
manager form of the client so resources are cleaned up immediately
without relying on refcounting or garbage collection.
MozReview-Commit-ID: HRSBDlYgqpC
--HG--
extra : rebase_source : 83deb56a0c2efefa883d6df104cd67194a811907
If configure has defined VCS binaries, we should use those.
MozReview-Commit-ID: DVnsSaJC8eN
--HG--
extra : rebase_source : 6a940b6b4f7986eece350ab692242701dfbf41dd
Some callers already have a build config object. Let's not
make them call a function that imports buildconfig.
MozReview-Commit-ID: J22HhyVma9y
--HG--
extra : rebase_source : fe5151925aee76508d26a9368e7a2ebec7e23ddb
This allows .flake8 files to override one another, and fixes a pretty bad known
bug with our flake8 implementation. For example, say we have a .flake8 file at:
/foo/.flake8
Before this patch, if we ran |mach lint foo/bar|, the configuration defined in
that .flake8 file wouldn't get picked up. It would only work if running the
specific directory that contains it, e.g |mach lint foo|.
This change additionally allows multiple .flake8 files to be used. So if
there's one defined at both:
/.flake8
/foo/.flake8
Then running |mach lint foo/bar| will first apply the root .flake8, then the
one under /foo (overriding earlier configuration).
This bug still doesn't make flake8 configuration perfect though. Any directory
containing a .flake8 file still needs to be explicitly listed in the "include"
section of /tools/lint/flake8.yml. Otherwise in the example above, if running
|mach lint /|, it wouldn't be able to find /foo/.flake8. This is a hard problem
and is likely best solved by fixing flake8's upstream configuration handling.
Unfortunately this means we still can't switch from a whitelist to a blacklist.
MozReview-Commit-ID: 3DZAi1QHYYo
--HG--
extra : rebase_source : 51298c5847f6c2792581d9b312c87b70fa716ee1
The install manifest with the .track files uses os.path.exists() to
determine if a previously tracked file is no longer installed and needs
to be removed from the system. However, exists() returns False for
broken symlinks, so as far as the manifest is concerned, there is no
file in the filesystem that needs to be removed. We should use lexists()
so we know that the broken symlink still exists in the system so
that it can be removed when the install manifest is processed.
MozReview-Commit-ID: 6v7CYOKzjGs
--HG--
extra : rebase_source : 8aeeef59e644613f34c8458bd30a83d8299585ea
There's a persistent test failure in automation that seems to have to do
with shutting down the `multiprocessing.Manager` that's used to get a
`Queue` to submit jobs to worker processes. After toying around with fixing
that I decided it would be simpler to just use concurrent.futures here,
since we already have it in-tree and it fits the use case here better
than using raw multiprocessing.
MozReview-Commit-ID: 8DdSvs2qp0q
--HG--
extra : rebase_source : 0b1cbb96bd3016778e4974a311722a8882f87216
Before, the "relevant" moz.build files were based strictly on filename.
In reality, there are some moz.build files that we wish to ignore.
The previous commit introduced a Finder that knows how to ignore
moz.build files that should be ignored. In this commit, we hook
it up to our low-level function for determining the set of relevant
moz.build files for a path.
The main benefit of this change is that paths in the moz.build
test directory no longer say test moz.build files are relevant.
Previously, we would return these test moz.build files. Some of
these are invalid and would cause execution to fail. So, commands
like `mach file-info` will no longer attempt to evaluate moz.build
files they weren't supposed to and will stop erroring.
Another benefit is that the function returns faster. When passing
in every file in the repo (>230,000 files), execution time dropped
from ~8.03s to ~6.16s. This is probably due to fewer path operations.
MozReview-Commit-ID: J2d25ZtxjFt
--HG--
extra : rebase_source : 26105de85c49e061a720b54019b4f16b6425748e
Not every moz.build file in the repo is a normal moz.build file.
Some moz.build files are used for testing moz.build files. Others
may exist in directories that should be ignored.
all_mozbuild_path() already knew how to filter out moz.build files
that should be ignored. Let's extract the Finder for doing this
into an instance attribute so it can be used elsewhere.
MozReview-Commit-ID: 9PaZQAbjIZO
--HG--
extra : rebase_source : 5bfd27b5a9ab6b24b9e3aa3c8cc286d64c3ebd3c
This passes `python3 -mcompileall`. Changes:
* use `0o` prefix for octal literals
* print as a function
* except .. as
* use six.reraise to replace a multi-argument raise statement
* use six.string_types and six.moves.configparser
* remove uses of `L` suffix for long integers
MozReview-Commit-ID: KLaYRNHGpay
--HG--
extra : rebase_source : 6ca1b5447cd28eff8d9f2805add6a0f07e8b4c63
Bug 1355661 added support for brotli streams in "jar" files handled by
Gecko, and bug 1355671 made us build the `bro` command line utility that
allows to compress and decompress brotli streams.
This change uses the `bro` command line utility in the packager so that
it can create and handle "jar" files using brotli streams.
However, the `bro` command line utility is not available to l10n
repacks. As, at the moment, we're only hoping that the outcome of using
brotli will be good, we avoid doing all the work to make those work and
just hook things enough to enable brotli, while ensuring l10n repacks
don't break. This involves forcing some files to be deflated, and to
disable some optimizations from the packager.
Things will need to be figured out more properly if the experiment
proves brotli to be worthwhile.
--HG--
extra : rebase_source : a2e0cff67dcaed465fd441ed5d2a7de94b6351c5
Change webextensions experiments test to use the shimmed certficiate DB
instead of the extensions.legacy.enabled pref.
In builds that don't honor the extensions.legacy.enabled pref, disable
test_legacy.js since that tests that flipping that preference works properly.
Finally, remove a now doubly-obsolete test of plugins embedded in xpis.
MozReview-Commit-ID: JiRdgCXyjKR
--HG--
extra : rebase_source : f0c7672b0755993bd20f9fc84e242eb76cb949ef
This preserves ./mach settings' --list option. If --list is passed in, we call splitlines()
on the description and print only the first line. The full multi-line description will be
printed otherwise.
This also displays the type and/or choices of the option as appropriate.
MozReview-Commit-ID: 7UMsN9qslWt
--HG--
extra : rebase_source : 4bc9554d8652e02e290c6a190634f1a72cdbadc3
We've been recording the commit id from the last vendor in
README_MOZILLA inside the various media directories. Since
we now support a --repo switch to pull from forks, record
this info as well, to make it easier to find contiguous
upstream source.
MozReview-Commit-ID: 1RanpkWfAeC
--HG--
extra : rebase_source : b6bd16b56626a871802822385be6f3a24db6cd50
Add a --repo switch to `mach vendor aom` to allow specifying
an alternate repository url.
Update our vendor script to support commit query and
snapshot download from github as well as upstream's
gitiles instance.
This lets us work with experimental branches for testing.
Also cleans up some naming and checks the passed url for
one of the two supported sites. We could fall back to
doing a complete clone and query the local repository,
but this covers most use cases.
--HG--
extra : rebase_source : 4ecc095db4539b86de4e82a853d5b28ac66c7f1d
This is a quick and dirty hack to get treeherder to show pytest failures. Long term, we might
want to investigate using something like pytest-mozlog. But the benefit of this approach is
we get to keep pytest's fantastic default log format.
MozReview-Commit-ID: Gcsz6z8MeOi
--HG--
extra : rebase_source : 00ee7973eadf86c081b548d5e79c48ca951e25a6
``print()`` has no business being in library code like this. It was
a holdover from this code being copied from bootstrap. So remove it.
While we're here, replace the generic exception with a specific one.
We don't want to be swallowing bugs via ``except Exception``.
MozReview-Commit-ID: 49goUstfPBz
--HG--
extra : rebase_source : a821159bd12a449ed1a0edf21a1f9eb29711ad95
We currently raise if we detect a command has both the `parser` attribute and a subcommand
at the same time, but as far as I can tell, there's no good reason to do this. Handling a
parser + subcommands seems to work exactly how you would expect. Furthermore, it isn't an
error to have subcommands + @CommandArgument, so it doesn't make sense that we're allowing
one but not the other.
This change solves an (admittedly unique) use case I'm trying to build into |mach try|. There
are N subcommands that all support a --save and --load style argument. So, e.g, we might have:
./mach try syntax --save foo
./mach try fuzzy --save bar
The main command will have the ability to detect which subcommand a saved value was generated
from and automatically dispatch to it. So this will work:
./mach try --load foo # dispatches to the syntax subcommand
./mach try --load bar # dispatches to the fuzzy subcommand
In order to share the --save/--load arguments across the main command + subcommands, we need
to set the parser attribute.
MozReview-Commit-ID: KmXRj8TBvYK
--HG--
extra : rebase_source : cbf1e402a080913709a34430274ae3191821dd72
This will facilitate handling this error condition specially.
Some unused imports were also removed.
MozReview-Commit-ID: 4zxVDgE7NxU
--HG--
extra : rebase_source : c4d0dd96629e028ce8a83215564a8d865e4b4b3d
The Repository interface gains a new method to determine if a sparse
checkout is present. Mercurial's implementation is somewhat crude,
but it should work (Mercurial's sparse support is still experimental
and I only intend to support sparse checkout in Firefox CI until it
is less experimental). Git's always returns False (for now).
To prove it works and to expose the information more widely, we hook
it up to moz.configure. We do this by first implementing a function
that returns a Repository instance. Then we simply call a function on
it to resolve the sparse checkout flag.
MozReview-Commit-ID: AlsT5LdSPdZ
--HG--
extra : rebase_source : f1e9aaa7d15f11c7c5e8d268d4ad82468732103b
This will remove the need to sniff class types. The 1 in-tree
consumer doing this has been converted.
MozReview-Commit-ID: I8cUa8J54VE
--HG--
extra : rebase_source : 4c24adaf7eb9d62678ac78604e819a7376d4073b
The --track flag provides a more accurate accounting of what files were
installed by the manifest, so they can be appropriately removed. For
example, test files are now removed from _tests if an entry in a test
file is deleted.
The --no-remove flag is removed as an alternative, and the --track flag
is now mandatory.
MozReview-Commit-ID: Wiup4Gzwkb
--HG--
extra : rebase_source : 4a44c7fe066ba9b5f1e37ec682464f7f4f6cb2cf
This patch removes the ability to select which protocols you want
included in necko, a wholly untested configuration that is broken in
practice. We have no need of this kind of configurability in necko.
In addition, this removes the final vestiges of rtsp support, which was
originally removed in bug 1295885 but still had some stuff hanging
around behind some ifdefs (that were never true).
MozReview-Commit-ID: KOEaDmit2IL
--HG--
extra : rebase_source : f6c2fdb972aaba46e922cda801252dc953550b94
This switches most tests over to use pytest as the runner instead of unittest (taking
advantage of the fact that pytest can run unittest based tests).
There were a couple tests that had failures when swithing to pytest:
config/tests/unit-expandlibs.py
xpcom/idl-parser/xpidl/runtests.py
For these tests, I added a runwith='unittest' argument so that they still run the
same way as before. Once we fix them to use pytest, the unittest logic in mozunit.py
can be deleted.
MozReview-Commit-ID: Gcsz6z8MeOi
--HG--
extra : rebase_source : 3c762422ce0af54cbbe7d9fc20085a2d1ebe7057
Andrew Halberstadt
Wes Kocher
Zibi Braniecki
Tom Ritter
Philip Jenvey
Dustin J. Mitchell
Phil Ringnalda
Chris Manchester
Andi-Bogdan Postelnicu
Jan Beich
Steve Armand
SingingTree
Botond Ballo
Sebastian Hengst
Kris Maglione
Gregory Szorc
Mike Conley
Nick Fitzgerald
Gabriele Svelto
Jim Chen
Ryan VanderMeulen
Henri Kemppainen
Mike Shal
Johan Lorenzo
Ted Mielczarek
Mike Hommey
Chung-Sheng Fu
Andrew Swan
Ralph Giles
Tom Prince
Jeff Gilbert
Rok Garbas
Nicholas Hurley