This essentially tests tools/lint/python/flake8.py. Though it also adds
a basic framework for testing all the other linters as well. Getting this
added now will allow others to collaborate on adding more tests without
needing to get to 100% coverage for all linters right off the bat.
All python tests under tools/lint/test will run as part of the 'ml' task
on Linux, and the build task on Windows (OSX coverage is currently missing
for python tests).
The flake8 linter currently has a bug where the 'exclude' argument is
ignored. This is why we need to also exclude 'tools/lint/test/files' in
topsrcdir/.flake8, even though it is already listed in the
'mach_commands.py'. Other linters shouldn't need to do this, the exclusion
in 'mach_commands.py' should be good enough. See bug 1277851 for more
details.
MozReview-Commit-ID: 9ho8C83eeuj
--HG--
extra : rebase_source : 0adb01aa81c06f26cb891ae9f92a17f8f9fdf111
There a few pieces needed here to properly handle KeyboardInterrupts.
1. All in-progress work needs to abort. Ideally the underlying linters will be
able to catch KeyboardInterrupt, and return partial results (like the flake8
linter does). Linters may alternatively allow the KeyboardInterrupt to
propagate up. Mozlint will catch and handle this appropriately, though any
results found will be lost. The only change to this behaviour was fixing a bug
in the flake8 linter.
2. Any unstarted jobs need to be canceled. In concurrent.futures, there are two
different queues. First, jobs are placed on the work queue, which is just a list
maintained by the parent process. As workers become available, jobs are moved
off the work queue, and onto the call queue (which is a multiprocessing.Queue).
Jobs that live on the work queue can be canceled with 'future.cancel()', whereas
jobs that live on the call queue cannot. The number of extra jobs that are stored
on the call queue is determined by this variable:
https://hg.mozilla.org/mozilla-central/file/deb7714a7bcd/third_party/python/futures/concurrent/futures/process.py#l86
In this patch, the parent process' sigint handler (which will be called on Ctrl-C)
is responsible for canceling all the jobs on the work queue. For the jobs on the
call queue, the best we can do is set a global variable that tells workers to
abort early.
3. Idle workers should exit gracefully. When there are no more jobs left, workers
will block on the call queue (either waiting for more jobs, or waiting for the
executor to send the shutdown signal). If a KeyboardInterrupt is received while a
worker is blocking, it isn't possible to intercept that anywhere (due to quirks
of how concurrent.futures is implemented). The InterruptableQueue class was
created to solve this problem. It will return None instead of propagating
KeyboardInterrupt. A None value will wake the worker up and tell it to gracefully
shutdown. This way, we avoid cryptic tracebacks in the output.
With all of these various pieces solved, pressing Ctrl-C appears to always exit
gracefully, sometimes even printing partial results.
MozReview-Commit-ID: 36Pe3bbUKmk
--HG--
extra : rebase_source : d4c312ee5cc3679eeee1407c5521aed679f84ad4
extra : source : a93a00141bf62f6bc9e30934c0e56f6b2e434bf0
This fixes a bug which can happen when the default version of python differs from
the version of python used with mach.
For example, mach explicitly looks for python2.7. This means running |mach lint -l flake8|
should also run flake8 with version 2.7. But if the default is python3, and flake8 is also
installed there, the subprocess call that invokes flake8 will run under python3. This can
lead to errors like "undefined name 'basestring'" and other 2to3 gotchas.
This patch ensures that we run the flake8 binary (and the pip for installing flake8) from
the same interpreter as mach, no matter the system default.
MozReview-Commit-ID: HSuMzDsAvsW
--HG--
extra : rebase_source : faf4c8c0eb6f46d8b50c2d9be9aa6f6d47e4e0cb
This patch makes a few changes around error handling:
1) Prints the name of the linter that produced non-json output
2) Changes the 'python not found' error to a warning (as this is not fatal)
3) Makes sure said warning only gets printed once (by moving it to the setup function)
MozReview-Commit-ID: Dkq7CulTs91
--HG--
extra : rebase_source : 5d4bd32a62264a88520c09420f5acd90edcdc740
This fixes a bug which can happen when the default version of python differs from
the version of python used with mach.
For example, mach explicitly looks for python2.7. This means running |mach lint -l flake8|
should also run flake8 with version 2.7. But if the default is python3, and flake8 is also
installed there, the subprocess call that invokes flake8 will run under python3. This can
lead to errors like "undefined name 'basestring'" and other 2to3 gotchas.
This patch ensures that we run:
python2.7 -m flake8
which explicitly runs flake8 against the same interpreter as mach, no matter the default.
MozReview-Commit-ID: HSuMzDsAvsW
--HG--
extra : rebase_source : d39f97ed0f82f4d2c81f044ef1f7dad0953a3d8d
This allows linters to define a 'setup' method which will automatically be
called by |mach lint| before running the linter. Users can also explicitly run
these methods (without doing any actual linting) by running |mach lint --setup|.
MozReview-Commit-ID: 74aY1pfsaX1
--HG--
extra : rebase_source : e6a7d769ba14c996c7a77316928063fa46358c5a
The flake8 linter requires a specific version of flake8, so that running it
locally vs in CI vs in mozreview will all produce the same results.
Allowing consumers to specify a custom flake8 binary via the FLAKE8 environment
variable, subverts that goal and can result in unexplained errors showing up in
some configurations but not others.
MozReview-Commit-ID: 1s0nC8ZO6Qi
--HG--
extra : rebase_source : f3913e2bce7770edf8deba18890f063975ca0844
The excluded directories aren't being properly handled in the py2/py3 compat
linters. In order for FileFinder to apply the exclusions properly they need
to either be relative to or contained by the base.
This means that currently the following will work:
./mach lint -l py2 <topsrcdir>
./mach lint -l py2 testing/mochitest
But this is broken:
./mach lint -l py2 testing
This change fixes the compat linters so exclude paths will be made relative
to the FileFinder base before passing them in. Any exclude not contained by
the base is simply discarded as it won't be relevant to that FileFinder
instance anyway.
MozReview-Commit-ID: LJx97TvKlSa
--HG--
extra : rebase_source : b8f0cb20ec5a88a33c26ace699ed9216b070f443
check_compat.py was adapted from gps' check-py3-compat.py in mercurial:
https://www.mercurial-scm.org/repo/hg/file/tip/contrib/check-py3-compat.py
The py3 linter simply runs ast.parse(f) for each file being linted. Any syntax errors
are formatted as mozlint results and dumped to stdout as json. I looked into also
importing the file (using 3.5+'s importlib.util.spec_from_file_location), but there
were too many problems:
1. Lots of false positives (e.g module not found)
2. Some files seemed to run indefinitely on import
I decided to punt on importing for now, we can always investigate in a follow-up.
The py2 linter runs ast.parse(f), and also checks that the file has:
from __future__ import absolute_import, print_function
Initially every python file in the tree is excluded from the py2 check, though
at least this makes it easy to find+fix, and new files in un-excluded
directories will automatically be linted.
MozReview-Commit-ID: ABtq9dnPo9T
--HG--
extra : rebase_source : 60762937284d498514cd020b90cbfd2ba23f0b70
This will allow us keep python related linting files in the same place.
MozReview-Commit-ID: ABtq9dnPo9T
--HG--
rename : tools/lint/flake8_/__init__.py => tools/lint/python/__init__.py
rename : tools/lint/flake8_/__init__.py => tools/lint/python/flake8.py
rename : tools/lint/flake8_/flake8_requirements.txt => tools/lint/python/flake8_requirements.txt
extra : rebase_source : 2568bc0bf8f4adbf8e0be73a54d5da068a8d81b0
Andrew Halberstadt
Noemi Erli