When using globs in exclude directorives, FileFinder will return every *file*
that gets matched. This is can be thousands of files in the case of an objdir.
While we now collapse these files down to highest possible directories, this
collapse operation can still take a noticeable amount of time (0.6s). This
simply scans topsrcdir for files that start with 'obj' to avoid the glob.
This also moves the '_activate_virtualenv' call to the top of the function
because in CI, this will cause an objdir to be created (to store the
virtualenv). If this happens *after* calculating the global excludes, we won't
catch it since it doesn't exist yet. This will result in the objdir's
virtualenv being linted and erroneous failures.
Depends on D7739
Differential Revision: https://phabricator.services.mozilla.com/D7740
--HG--
extra : moz-landing-system : lando
Often we specify globs in our exclude patterns, e.g:
exclude:
- **/node_modules
- obj*
However, these globs get expanded out to *every* file that matches them. This
can sometimes be thousands or even tens of thousands of files.
We then pass these paths on to the underlying linters and tell them to
exclude them all. This causes a lot of overhead and slows down performance.
This commit implements a "collapse" function. Given a set of paths, it'll
collapse them into the smallest set of parent directories that contain the
original set, and that don't contain any extra files.
For example, given a directory structure like this:
a
-- foo.txt
-- b
-- bar.txt
-- baz.txt
-- c
-- ham.txt
-- d
-- spam.txt
Then the following will happen:
>>> collapse(['a/foo.txt', 'a/b/bar.txt', 'a/c/ham.txt', 'a/c/d/spam.txt'])
['a/foo.txt', 'b/bar.txt', 'c']
Since all files under directory 'c' are specified by the original set (both
'c/ham.txt' and 'c/d/spam.txt'), we can collapse it down to just 'c'. However
not all files under 'b' are specified (we're missing 'a/b/baz.txt'), so we
can't collapse 'b' (and by extension also can't collapse 'a').
If we had included 'a/b/baz.txt':
>>> collapse(['a/foo.txt', 'a/b/bar.txt', 'a/b/baz.txt', 'a/c/ham.txt', 'a/c/d/spam.txt'])
['a']
In both cases, the smallest set of paths that contains the original set (and
only the original set) is computed.
The collapse function has a little bit of overhead but it's not too bad.
For example collapsing all files matched by '**/node_modules' takes ~0.015s.
Collapsing two full objdirs, takes ~0.6s. But a follow up commit is planned to
make sure we stop using 'obj*' to reduce that overhead.
Depends on D7738
Differential Revision: https://phabricator.services.mozilla.com/D7739
--HG--
extra : moz-landing-system : lando
If scandir is already present on the system the attempt to import the
c helper library will currently find the c helper from the system
install which may well be an outdated verion, so causing mach to
break. To solve this this patch does two things:
* Stops importing scandir in files that are run unconditionally when
invoking mach. This is generally considered good for performance
reasons.
* Installs the vendored scandir into the virtualenv for `mach lint`
rather than trying to import it directly from the source tree and so
not getting the c helper library.
Differential Revision: https://phabricator.services.mozilla.com/D8379
--HG--
extra : moz-landing-system : lando
When using globs in exclude directorives, FileFinder will return every *file*
that gets matched. This is can be thousands of files in the case of an objdir.
While we now collapse these files down to highest possible directories, this
collapse operation can still take a noticeable amount of time (0.6s). This
simply scans topsrcdir for files that start with 'obj' to avoid the glob.
This also moves the '_activate_virtualenv' call to the top of the function
because in CI, this will cause an objdir to be created (to store the
virtualenv). If this happens *after* calculating the global excludes, we won't
catch it since it doesn't exist yet. This will result in the objdir's
virtualenv being linted and erroneous failures.
Depends on D7739
Differential Revision: https://phabricator.services.mozilla.com/D7740
--HG--
extra : moz-landing-system : lando
This essentially tests tools/lint/python/flake8.py. Though it also adds
a basic framework for testing all the other linters as well. Getting this
added now will allow others to collaborate on adding more tests without
needing to get to 100% coverage for all linters right off the bat.
All python tests under tools/lint/test will run as part of the 'ml' task
on Linux, and the build task on Windows (OSX coverage is currently missing
for python tests).
The flake8 linter currently has a bug where the 'exclude' argument is
ignored. This is why we need to also exclude 'tools/lint/test/files' in
topsrcdir/.flake8, even though it is already listed in the
'mach_commands.py'. Other linters shouldn't need to do this, the exclusion
in 'mach_commands.py' should be good enough. See bug 1277851 for more
details.
MozReview-Commit-ID: 9ho8C83eeuj
--HG--
extra : rebase_source : 0adb01aa81c06f26cb891ae9f92a17f8f9fdf111
Changed flake8 installation strategy, try to update it using testings/lint/flake8_requirements.txt
moved flake8_requirements to tools/lint/flake8 directory
extracted common pip invocations to _run_pip function
minor fixes
fixed calls to check_process (concatenating list to tuple)
removed update_pip function and its error message variable
if flake8 installation reports and error print pip command output - replaced out with e.output in print function
MozReview-Commit-ID: Fmq22aL8dMq
--HG--
extra : rebase_source : 2e2f942c1ffab012795c632ceacf6debf0dc1407
This integrates 'eslint' with the mozlint framework. The old |mach eslint| command is kept around
for backwards compatibility and will simply dispatch to |mach lint|. But |mach lint| should be
preferred as the old command may eventually be removed.
The old |mach eslint| command should be mostly backwards compatible with a few exceptions:
1. Can no longer define --extensions on the command line, this is instead hardcoded into
eslint.lint.
2. No longer using eslint formatters. However, the default mozlint formatter should be
identical to the default eslint formatter, so developers should not notice a change.
This does mean that non-default eslint formatters can no longer be used unless a mozlint
copy of them is created.
3. Installs dependencies automatically without prompting the user. This was necessary due to
python multiprocessing limitations, but is actually also a better UX. Because the npm
dependencies aren't global anymore, there isn't really any reason *not* to install them
automatically.
Apart from that, any difference from the old |mach eslint| I'd consider a bug.
The main eslint implementation now lives in tools/lint/eslint.lint instead of
tools/lint/mach_commands.py.
MozReview-Commit-ID: KYhC6SEySC3
--HG--
extra : rebase_source : 36d06b1a3fced764e17154cc53091d7722919b67
This removes the in-tree plugin from the tooltool archive and uses that code
directly from the Gecko checkout instead.
For automation, we now get ESLint and external plugins from tooltool and then
symbolic link to the in-tree plugin.
For local development, we install ESLint and external plugins following the
shrinkwrap file created from the last change to the tooltool archive. The local
plugin is then installed.
This change also removes the list of module versions from mach_commands.py, so
there is only one place to update module versions for the future.
MozReview-Commit-ID: AhbZ8lVPmN4
The file base/content/browser-fullScreen.js was renamed. This patch updates the reference
to this filename in the esling rule import-browserjs-globals.js and bumps the version to
0.1.1.
MozReview-Commit-ID: 9hibfDnbtp
This adds two parameters, --rev and --workdir. Each works both with mercurial and git (though the syntax for
specifying revisions is different between them). The value is simply forwarded to either |hg log| or |git diff|
so syntax like |mach lint -r .~4::.| or |mach lint -r "HEAD~4 HEAD"| will work as expected.
MozReview-Commit-ID: aOGp2Yrncs
--HG--
extra : rebase_source : d2cb834d4cc1a083171a3551af4e72c8a7d14021
There is currently no built-in user interface to mozlint. The only existing interface is the
external cli provided by |mach lint|. However, in the future mozlint may need to be used in a
context where mach isn't readily available (i.e version-control-tools). This patch basically
just moves the cli logic out of mach_commands.py, and into mozlint core. That way it can be
re-used in other places without needing to be re-implemented.
The |mach lint setup| subcommand was removed because apparently subcommands don't work with
the parser attribute. Nothing was using it yet anyway, so I removed it for now. It may get
re-added in some form in the future.
MozReview-Commit-ID: aOGp2Yrncs
--HG--
extra : rebase_source : 8f7530de96e5c131d2ed5bfcdd7a159329401e5b
This makes it easier to lint a path that otherwise wouldn't have been linted due to the include/exclude
directives. Now, you can pass in -n/--no-filter instead of needing to modify the linter configuration file.
MozReview-Commit-ID: GMJuE2C1NyY
--HG--
extra : rebase_source : 03627e930f76903ad629cb01b58c4ae7372e4bb1
Enables flake8 linting! To start, only these directories are actually linted:
- python/mozlint
- tools/lint
To enable new directories, add them to the 'include' directive at the bottom of:
tools/lint/flake8.lint
Edit topsrcdir/.flake8 to modify global configuration. Add a new .flake8 to a
subdirectory to override the global. The current configuration is more or less
just the default and we should tweak it to our needs.
MozReview-Commit-ID: iXbToRhm3b
--HG--
extra : transplant_source : %E7i%04%07%1C%B3%E4%FD%FF%BB%A6w%CF%27t%FFz%CD%5EE
Enables flake8 linting! To start, only these directories are actually linted:
- python/mozlint
- tools/lint
To enable new directories, add them to the 'include' directive at the bottom of:
tools/lint/flake8.lint
Edit topsrcdir/.flake8 to modify global configuration. Add a new .flake8 to a
subdirectory to override the global. The current configuration is more or less
just the default and we should tweak it to our needs.
MozReview-Commit-ID: iXbToRhm3b
--HG--
extra : rebase_source : 78c4491261ba36421110b0d9fca73c1f6d1f85c3
For now, only the following two directories will be linted:
python/mozlint
tools/lint
New directories can be added by adding them to the 'include'
directive in tools/lint/flake8.lint. They all default to the
configuration specified in topsrcdir/.flake8. Subdirectories
can override this configuration by creating their own .flake8
file.
MozReview-Commit-ID: Eag48Lnkp3l
--HG--
extra : rebase_source : 6d98c9fef80055a48cc2622848aa04c3045b747e
Andrew Halberstadt
Csoregi Natalia
Narcis Beleuzu
James Graham
Tom Prince
Mark Banner
Carsten "Tomcat" Book
Francesco Pischedda
Philipp Kewisch
Matthew Noorenberghe
Jared Wein
J. Ryan Stinnett
Tom Tromey
Julian Descottes