We were currently adding exclude paths to the "discard" set if the path contains
the include, but we weren't adding them if the include contains the path.
Depends on D5863
Differential Revision: https://phabricator.services.mozilla.com/D8845
--HG--
extra : moz-landing-system : lando
Right now there are excludes defined in the linter definition (via the .yml
files), as well as excludes defined in lintargs (via the mach_commands.py).
This is a minor simplification that extends each linter definition's local
excludes with the global ones right off the bat. This just makes it a bit
easier to keep track of.
Depends on D5863
Differential Revision: https://phabricator.services.mozilla.com/D8844
--HG--
extra : moz-landing-system : lando
Currently pathutils.filterpaths computes some extra paths that the underlying linter
should try to exclude if it can. It sets these on lintargs['exclude'], and relies on
the fact that it was passed by reference to work.
However, it seems like pytest does some magic under the hood that prevents this value
from being propagated back. It will also fail if 'filterpaths' was invoked in a
subprocess.
It's much simpler and cleaner to pass this value back directly, rather than rely on a
reference.
Differential Revision: https://phabricator.services.mozilla.com/D8850
--HG--
extra : moz-landing-system : lando
This fixes a latent bug that is currently not being hit (by sheer luck). Basically
the 'ignore' argument of a FileFinder object needs all paths to be relative to the
base. As luck would have it, most of the time it worked out that way if you were
running |mach lint| from the root of the repo.
However there are edge cases where this will cause an 'exclude' directive to get
missed. Plus this bug is about to be exposed 100% of the time in the next commit :).
Depends on D8842
Differential Revision: https://phabricator.services.mozilla.com/D8843
--HG--
extra : moz-landing-system : lando
I missed an edge case where the computed base itself was specified in the
paths input.
Differential Revision: https://phabricator.services.mozilla.com/D8842
--HG--
extra : moz-landing-system : lando
This patch also changes how pdbs for the ASAN job are copied:
we relax restrictions so that pdbs if present) are always copied out
and add an environment variable MOZ_COPY_PDBS to indicate when we
want to produce pdbs for copying.
The desired outcome of this change is that we'll set
`-Wl,--version-script` based on linker kind and not on the output of
`$LINKER -v`.
This is a cheap way to address a simple problem that has a complicated
ideal solution. The underlying issue is that in some situations, when
targeting Android, a macOS system `ld` is interrogated to determine if
a cross-compiling linker "is GNU ld" and a particular linker feature
is set in that situation. The macOS system `ld` doesn't pass the "is
GNU ld" test, and the linker feature isn't set; that causes link
failures, even though the actual linker has nothing to do with the
system `ld`.
The ideal solution is to test for linker capabilities dynamically. We
do a lot of that in old-configure.in, and we don't do any of that in
toolchain.configure. Rather than start testing in
toolchain.configure, we hard-code: a cheap solution to the immediate
problem.
Differential Revision: https://phabricator.services.mozilla.com/D8471
--HG--
extra : moz-landing-system : lando
While we do have some uses of @depends-function comparison in some
templaces, related to host/target, we ought to be using `is` comparisons
rather than `==` anyways, so we switch those, and prevent other kinds of
comparisons being used at all.
This unveils the one noted in
https://phabricator.services.mozilla.com/D7713?id=21357#inline-30414
(and surprisingly only that one), that we remove entirely since it was
doing nothing in practice. Bug 1492305 will have to add it back in a
proper form.
Differential Revision: https://phabricator.services.mozilla.com/D8501
--HG--
extra : moz-landing-system : lando
Often we specify globs in our exclude patterns, e.g:
exclude:
- **/node_modules
- obj*
However, these globs get expanded out to *every* file that matches them. This
can sometimes be thousands or even tens of thousands of files.
We then pass these paths on to the underlying linters and tell them to
exclude them all. This causes a lot of overhead and slows down performance.
This commit implements a "collapse" function. Given a set of paths, it'll
collapse them into the smallest set of parent directories that contain the
original set, and that don't contain any extra files.
For example, given a directory structure like this:
a
-- foo.txt
-- b
-- bar.txt
-- baz.txt
-- c
-- ham.txt
-- d
-- spam.txt
Then the following will happen:
>>> collapse(['a/foo.txt', 'a/b/bar.txt', 'a/c/ham.txt', 'a/c/d/spam.txt'])
['a/foo.txt', 'b/bar.txt', 'c']
Since all files under directory 'c' are specified by the original set (both
'c/ham.txt' and 'c/d/spam.txt'), we can collapse it down to just 'c'. However
not all files under 'b' are specified (we're missing 'a/b/baz.txt'), so we
can't collapse 'b' (and by extension also can't collapse 'a').
If we had included 'a/b/baz.txt':
>>> collapse(['a/foo.txt', 'a/b/bar.txt', 'a/b/baz.txt', 'a/c/ham.txt', 'a/c/d/spam.txt'])
['a']
In both cases, the smallest set of paths that contains the original set (and
only the original set) is computed.
The collapse function has a little bit of overhead but it's not too bad.
For example collapsing all files matched by '**/node_modules' takes ~0.015s.
Collapsing two full objdirs, takes ~0.6s. But a follow up commit is planned to
make sure we stop using 'obj*' to reduce that overhead.
Depends on D7738
Differential Revision: https://phabricator.services.mozilla.com/D7739
--HG--
extra : moz-landing-system : lando
If scandir is already present on the system the attempt to import the
c helper library will currently find the c helper from the system
install which may well be an outdated verion, so causing mach to
break. To solve this this patch does two things:
* Stops importing scandir in files that are run unconditionally when
invoking mach. This is generally considered good for performance
reasons.
* Installs the vendored scandir into the virtualenv for `mach lint`
rather than trying to import it directly from the source tree and so
not getting the c helper library.
Differential Revision: https://phabricator.services.mozilla.com/D8379
--HG--
extra : moz-landing-system : lando
At least one Python package that got vendored via `mach vendor python`
contains a symlink, which hg.mozilla.org rejects. This change makes it so
symlinks get replaced with the contents of the file they point to.
Differential Revision: https://phabricator.services.mozilla.com/D4135
--HG--
extra : moz-landing-system : lando
This patch adds two new actions to virtualenv_packages.txt processing:
windows and !windows. The former processes the rest of the action only on
Windows, and the latter processes it only on non-Windows.
These new features are used in virtualenv_packages.txt to use the
path to the unpacked Windows psutil wheel when on Windows, and build psutil
from source and use that path on other platforms.
This fixes the long-standing problem of not having psutil available on most
Windows systems (since they don't have the right set of Visual C++ build tools).
Differential Revision: https://phabricator.services.mozilla.com/D3437
--HG--
extra : moz-landing-system : lando
This option is very single-purpose: it's intended to let us vendor an unpacked
wheel for psutil on Windows. To that end the mach command will error if you
try to use it for anything but vendoring a single package. The mach command
will vendor source packages as it currently does, and then run `pip download`
again with some hardcoded parameters to fetch the right wheel for Python 2.7
on win64 and unpack it to a `package-platform` directory under
`third_party/python`.
I don't expect this to be used for anything but psutil, but it should make life
simpler for anyone that wants to update our vendored copy of psutil in the
future.
Differential Revision: https://phabricator.services.mozilla.com/D3435
--HG--
extra : moz-landing-system : lando
Often we specify globs in our exclude patterns, e.g:
exclude:
- **/node_modules
- obj*
However, these globs get expanded out to *every* file that matches them. This
can sometimes be thousands or even tens of thousands of files.
We then pass these paths on to the underlying linters and tell them to
exclude them all. This causes a lot of overhead and slows down performance.
This commit implements a "collapse" function. Given a set of paths, it'll
collapse them into the smallest set of parent directories that contain the
original set, and that don't contain any extra files.
For example, given a directory structure like this:
a
-- foo.txt
-- b
-- bar.txt
-- baz.txt
-- c
-- ham.txt
-- d
-- spam.txt
Then the following will happen:
>>> collapse(['a/foo.txt', 'a/b/bar.txt', 'a/c/ham.txt', 'a/c/d/spam.txt'])
['a/foo.txt', 'b/bar.txt', 'c']
Since all files under directory 'c' are specified by the original set (both
'c/ham.txt' and 'c/d/spam.txt'), we can collapse it down to just 'c'. However
not all files under 'b' are specified (we're missing 'a/b/baz.txt'), so we
can't collapse 'b' (and by extension also can't collapse 'a').
If we had included 'a/b/baz.txt':
>>> collapse(['a/foo.txt', 'a/b/bar.txt', 'a/b/baz.txt', 'a/c/ham.txt', 'a/c/d/spam.txt'])
['a']
In both cases, the smallest set of paths that contains the original set (and
only the original set) is computed.
The collapse function has a little bit of overhead but it's not too bad.
For example collapsing all files matched by '**/node_modules' takes ~0.015s.
Collapsing two full objdirs, takes ~0.6s. But a follow up commit is planned to
make sure we stop using 'obj*' to reduce that overhead.
Depends on D7738
Differential Revision: https://phabricator.services.mozilla.com/D7739
--HG--
extra : moz-landing-system : lando
By doing this in the packager, it makes it easier to incorporate the
strip and XZ compress logic into the local Gradle build process.
To that end, this patch makes XZ compression a little more explicit in
package-manifest.in and lifts the logic next to the existing logic for
stripping. Since we only want to XZ compress assets/ (and not libs/),
we need a new flag.
Differential Revision: https://phabricator.services.mozilla.com/D7314
--HG--
extra : moz-landing-system : lando
By doing this in the packager, it makes it easier to incorporate the
strip and XZ compress logic into the local Gradle build process.
To that end, this patch makes XZ compression a little more explicit in
package-manifest.in and lifts the logic next to the existing logic for
stripping. Since we only want to XZ compress assets/ (and not libs/),
we need a new flag.
Differential Revision: https://phabricator.services.mozilla.com/D7314
--HG--
extra : moz-landing-system : lando
In bug 1461714 we run node.py to generate main.js in the objdir, and use
FINAL_TARGET_FILES to install it into dist. However, when both rules are
run in parallel in the misc tier, the install target may pick up the
main.js file from the srcdir via VPATH, so we end up with the wrong file
in dist. This causes some mochitests to fail with "uncaught exception -
SyntaxError: import declarations may only appear at top level of a
module at
@resource://devtools/client/debugger/new/src/main.js:7:undefined"
The workaround here is to run these node.py invocations (which always
write to node.stub) in the export tier, so that when the install target
in misc is processed, the objdir version of the file is always present
and takes precedence.
A better fix would probably be to remove our reliance on VPATH, and just
pass in the path to files in the srcdir when they are required. That
could potentially be a major overhaul, however.
MozReview-Commit-ID: JZ04C7zJPbX
Differential Revision: https://phabricator.services.mozilla.com/D7770
--HG--
extra : moz-landing-system : lando
This fixes treatment of Windows paths, where tilde is not special,
particularly in short file names.
--HG--
extra : rebase_source : be1873a44cb2d3fa455e7bd4a31bb7c221f19213
The previous code split a lot of logic between the update and download
parts, and exposed two different mach commands. In order to simplify
things it's better to have a single entry point for manifest download
and update, and ensure that's called consistently.
Differential Revision: https://phabricator.services.mozilla.com/D7497
This is a fairly straightforward port of the AS tool checks from old-configure
to toolchain.configure. AS is a little quirky in that we currently do a
normal-looking check for it, but then override that value to be the C compiler
for non-Windows builds, and ml[64]/armasm64 for Windows builds.
After migrating those checks, the only things left in the MOZ_DEFAULT_COMPILER
macro in compiler-opts.m4 were some unused bits, so I removed them:
* Setting of CPP/CXXCPP, which are set in toolchain.configure now
* Setting HOST_LDFLAGS to empty, which doesn't seem particularly useful.
There was also a quirky old test that the assembler was ml[64] when js-ctypes
is enabled that I removed, I don't think it provides any value since this
patch will ensure that we're using the right assembler for Windows builds.
Behavior changes:
- instead of reading the mozconfig file, it now instantiates an instance of the build object.
- safe checking methods are used to access attributes to prevent errors on automation environment.
- better mach command parsing is performed with handler category instead of error-prone argv parsing.
Other changes:
- docstring for testing/xpcshell/runxpcshelltests.py::buildTestList() added and modernized.
- added clause that if length of tests gathered is 0, mach exits with an error code of 1.
Differential Revision: https://phabricator.services.mozilla.com/D7133
--HG--
extra : moz-landing-system : lando
The previous code split a lot of logic between the update and download
parts, and exposed two different mach commands. In order to simplify
things it's better to have a single entry point for manifest download
and update, and ensure that's called consistently.
Differential Revision: https://phabricator.services.mozilla.com/D7497
--HG--
extra : moz-landing-system : lando
blessings.tigetstr is not part of its API. It happens to work because
blessings imports curses using 'from curses import tigetstr'.
Instead, we can just use terminal.normal, which contains the string we were
going to get anyway.
See https://github.com/erikrose/blessings/pull/138 for more information.
Let me know if there's a better way of resolving this. Hopefully with this +
the patch I submitted to blessings (https://github.com/erikrose/blessings/pull/137)
firefox will build fine with TERM improperly set.
Differential Revision: https://phabricator.services.mozilla.com/D5377
--HG--
extra : moz-landing-system : lando
- added checkers in python/mach/mach/main.py prior to calling registrar.py.
- added internal function to check if specified debugger is installed.
- support both ./mach test <test_name> and ./mach <test_category> styles.
Differential Revision: https://phabricator.services.mozilla.com/D7234
--HG--
extra : moz-landing-system : lando
- added checkers in python/mach/mach/main.py prior to calling registrar.py.
- added internal function to check if specified debugger is installed.
- support both ./mach test <test_name> and ./mach <test_category> styles.
Differential Revision: https://phabricator.services.mozilla.com/D7234
--HG--
extra : moz-landing-system : lando
- moved placement of the raise_exception computation to be after the initial objdir path computation.
- original implementation was missing detection for cases where user may pass in a False flag to explicitly suppress errors.
- added unit tests to check for scenarios where raise_exception flag is passed in as part of positional argument.
- changed argument to include a **kwargs argument for caller to modify default exception behavior.
- default behavior is to raise exceptions if mozinfo.json cannot be found.
- disabled TreeMetadataEmitter from calling mozinfo.find_and_update_from_json and setting the self.info variable since it was not referenced anywhere else after the initial setup.
Depends on D6859
Differential Revision: https://phabricator.services.mozilla.com/D6860
--HG--
extra : moz-landing-system : lando
There is only a single linter (test-disable.yml) that uses a glob in any
include path, and that usage is easily replaced by using the 'extensions' key
instead.
Since globs in include directives aren't very useful, let's disallow them. This
will allow us to simplify the 'filterpaths' logic quite substantially and make
future refactorings in this area easier.
Differential Revision: https://phabricator.services.mozilla.com/D6798
--HG--
extra : moz-landing-system : lando
This extracts the current logic for finding nodejs into its own module in mozbuild. Configure and ESLint then use it.
For ESLint, this will change the first location it looks for nodejs to be the .mozbuild directory.
Differential Revision: https://phabricator.services.mozilla.com/D6430
--HG--
extra : moz-landing-system : lando
This makes things more explicit. Previously we were relying on those magic
global "linters" variables, and it turned out that one of the tests was
actually linting a completely different set of linters than I was expecting.
This changes things so each test needs to explicitly define which linters it
wants to use.
Depends on D6410
Differential Revision: https://phabricator.services.mozilla.com/D6412
--HG--
extra : moz-landing-system : lando
This makes this test match all the other tests (which are named after the module
they are testing).
Also rename the test function to 'test_filterpaths'.
Differential Revision: https://phabricator.services.mozilla.com/D6410
--HG--
rename : python/mozlint/test/test_filterpaths.py => python/mozlint/test/test_pathutils.py
extra : moz-landing-system : lando
This patch gets us building an updater binary that always embeds the dep certificates (instead of release or nightly), and builds a new tests package that includes it.
This was originally D5900, but that was backed out due to busting artifact builds. I've fixed that by removing the Makefile that Ted pointed out is unnecessary.
Differential Revision: https://phabricator.services.mozilla.com/D6418
--HG--
extra : moz-landing-system : lando
Migrates strings from aboutUrlClassifier.dtd and aboutUrlClassifier.properties files to aboutUrlClassifier.ftl
Modifies aboutUrlClassifier.xhtml and aboutUrlClassifier.js to use fluent.
Differential Revision: https://phabricator.services.mozilla.com/D5156
--HG--
extra : moz-landing-system : lando
Changes the wpt manifest path to the topobjdir instead so it can be moved out of tree.
Other changes so that the manifest download and update, and |mach wpt| and |mach test <wpt-test>| work with the new path.
The manifest is also downloaded and updated when creating the tests-archive to ensure that it exists when we run tests on TC.
MozReview-Commit-ID: Fp6UsKJjhTU
Differential Revision: https://phabricator.services.mozilla.com/D5312
--HG--
extra : moz-landing-system : lando
Add support for |with Files('a/**', 'b/**')| in mozbuild config files.
MozReview-Commit-ID: IoM4qfEhXXc
Differential Revision: https://phabricator.services.mozilla.com/D5315
--HG--
extra : moz-landing-system : lando
This is a quick fix to workaround linking in rkv and lmdb-sys in the tup
backend. Bug 1492291 is a followup to actually fix the problem more
generally.
MozReview-Commit-ID: 523it7WEUad
Differential Revision: https://phabricator.services.mozilla.com/D6285
--HG--
extra : moz-landing-system : lando
This patch gets us building an updater binary that always embeds the dep certificates (instead of release or nightly), and builds a new tests package that includes it.
I had a lot of trouble getting the test package generated correctly due to the fact that things in "_tests" won't be included for test packages that aren't "common". My fix for that isn't ideal - I'm open to something better.
Differential Revision: https://phabricator.services.mozilla.com/D5900
--HG--
extra : moz-landing-system : lando
Andrew Halberstadt
Dave Hunt
Nick Alexander
Dorel Luca
Tom Ritter
Cosmin Sabou
Mike Hommey
Csoregi Natalia
Chris Manchester
Narcis Beleuzu
James Graham
Andi-Bogdan Postelnicu
Qinghao_Jack_Song
Ciure Andrei
Ted Mielczarek
Noemi Erli
Mike Shal
Adam Gashlin
Tooru Fujisawa
Nathan Froyd
Edwin Gao
Raza Haider
Jay Kamat
Brindusan Cristian
shindli
Aaron Klotz
Haik Aftandilian
Paul Adenot
Dan Mosedale
Jim
Mark Banner
Bob Clary
Collin Wing
Ben Hearsum
Coroiu Cristina
Jack Smith
Ahilya Sinha
Ted Campbell
Bogdan Tara