Граф коммитов

129 Коммитов

Автор SHA1 Сообщение Дата
Andrew Halberstadt a8b5fc493f Bug 1433912 - [lint] Only run codespell linter on python/mozlint and tools/lint for now, r=sylvestre
This is a bustage fix that needs to get out quickly. Once landed we can take
the time to enable it on more directories, or preferably change it to a
blacklist.

MozReview-Commit-ID: Gbf7q2L0tg3

--HG--
extra : rebase_source : a58ae64c655b24e686710a663d4538b4cfe020f7
2018-01-29 10:25:54 -05:00
Andrew Halberstadt 529aad2e2c Bug 1430825 - [mozlint] Split work up by paths instead of by linters, r=standard8
The initial motivation for this patch, was to prevent command lines that are
too long on Windows. To that end, there is a cap to the number of paths that
can be run per job. For now that cap is set to 50. This will allow for an
average path length of 160 characters, which should be sufficient with room to
spare.

But another big benefit of this patch is that we are running more things in
parallel. Previously, mozlint ran each linter in its own subprocess, but that's
it. If running eslint is 90% of the work, it'll still only get a single
process. This means we are wasting cores as soon as the other linters are
finished.

This patch chunks the number of specified paths such that there will be N*L
jobs where 'N' is the number of cores and 'L' is the number of linters.  This
means even when there's a dominant linter, we'll be making better use of our
resources. This isn't perfect of course, as some paths might contain a small
number of files, and some will contain a very large number of files.  But it's
a start

A limitation to this approach is that if there are fewer paths specified than
there are cores, we won't schedule enough jobs per linter to use those extra
cores. One idea might be to expand specified directories and individually list
all the paths under the directory. But this has some hairy edge cases that
would be tough to catch. Doing this in a non-hacky way would also require a
medium scale refactor.

So I propose further parallelization efforts be destined for follow-ups.

MozReview-Commit-ID: JRRu13AFaii

--HG--
extra : rebase_source : 242fb71fe0af8bd2a981bd10a7216bb897fe00ac
2018-01-16 16:01:20 -05:00
Gurzau Raul e12c30afe7 Backed out changeset 5bb16f349a38 (bug 1430825) for Windows build bustage on a CLOSED TREE 2018-01-22 21:54:08 +02:00
Andrew Halberstadt 337dcdeb9d Bug 1430825 - [mozlint] Split work up by paths instead of by linters, r=standard8
The initial motivation for this patch, was to prevent command lines that are
too long on Windows. To that end, there is a cap to the number of paths that
can be run per job. For now that cap is set to 50. This will allow for an
average path length of 160 characters, which should be sufficient with room to
spare.

But another big benefit of this patch is that we are running more things in
parallel. Previously, mozlint ran each linter in its own subprocess, but that's
it. If running eslint is 90% of the work, it'll still only get a single
process. This means we are wasting cores as soon as the other linters are
finished.

This patch chunks the number of specified paths such that there will be N*L
jobs where 'N' is the number of cores and 'L' is the number of linters.  This
means even when there's a dominant linter, we'll be making better use of our
resources. This isn't perfect of course, as some paths might contain a small
number of files, and some will contain a very large number of files.  But it's
a start

A limitation to this approach is that if there are fewer paths specified than
there are cores, we won't schedule enough jobs per linter to use those extra
cores. One idea might be to expand specified directories and individually list
all the paths under the directory. But this has some hairy edge cases that
would be tough to catch. Doing this in a non-hacky way would also require a
medium scale refactor.

So I propose further parallelization efforts be destined for follow-ups.

MozReview-Commit-ID: JRRu13AFaii

--HG--
extra : rebase_source : 6cd73d8b6888723de3410df043f7ed042ba3349f
2018-01-16 16:01:20 -05:00
Andrew Halberstadt 0e697ce235 Bug 1422302 - Create python/mozterm for sharing terminal blessings across modules r=gps
This is a new module that will provide a place to store some common
abstractions around the 'blessings' module. The main entrypoint is:

    from mozterm import Terminal
    term = Terminal()

If blessings is available, this will return a blessings.Terminal()
object. If it isn't available, or something went wrong on import,
this will return a NullTerminal() object, which is a drop-in
replacement that does no formatting.

MozReview-Commit-ID: 6c63svm4tM5

--HG--
extra : rebase_source : 9ab221774d92a418d9b098d79bb2c88f75d937f8
2017-12-04 09:38:24 -05:00
Andreas Tolfsen 894334f11e Bug 1405304 - Add Unix formatter for mozlint. r=ahal
This patch introduces a new report formatter for the mozlint
framework used by "./mach lint" that respects Unix output conventions,
popularised by grep(1), compilers, and preprocessors.

The output format looks like this:

	testing/marionette/driver.js:1153:48: no-unused-vars error: 'resp' is defined but never used.

Many editors (ed, Emacs, vi, Acme) recognise this format, allowing
users to interact with the output like a hyperlink to jump to the
specified location in a file.

MozReview-Commit-ID: 3IyiFmNbtMY

--HG--
extra : rebase_source : a2628a999c187a1c43f3cc5d32e6db835028eca4
2017-10-03 14:45:17 +01:00
Andrew Halberstadt a73d388c79 Bug 1401309 - [mozlint] Remove vcs.py and use mozversioncontrol instead, r=gps
This also migrates the vcs.py test to mozversioncontrol and adds a new task for
it.

MozReview-Commit-ID: 9jTRkjNupVA

--HG--
extra : rebase_source : 400f27498e00ea45234ad7c951770b098e916b8e
2017-09-25 16:30:27 -04:00
Andrew Halberstadt d59f62ddf2 Bug 1392787 - [mozlint] Fix bugs in path filtering and add a test, r=jmaher
MozReview-Commit-ID: LG47ASBMA17

--HG--
extra : rebase_source : 3d6394d2839bf4c12dedbdb84f5b7dd118c92fea
2017-08-23 06:33:06 -04:00
Andrew Halberstadt dae5d6aae4 Bug 1399522 - [mozlint] Properly handle directories in LineLinters, r=bc
Currently line linters (linters that open a file and process it line by line,
by applying a regex for example), don't handle directories. If a directory is
passed in, it will try to 'open' it, which fails. Directories can get hit  if
the linter has a directory in its include directive or if the user passes in
--no-filter.

This patch modifies LineLinters so that if a directory is detected, we search
for all relevant files under that directory. If 'extensions' is used, we'll
look for only files with appropriate extensions. Otherwise we assume the
linter wants every file.

MozReview-Commit-ID: D9lzTNuQTob

--HG--
extra : rebase_source : 0b952c06eae28b67b687813ff7e75b231b2dd4d3
2017-09-13 12:03:18 -04:00
Steve Armand 08d5fcd935 Bug 1397423 - Enable py2 linter on python/mozlint. r=ahal
MozReview-Commit-ID: 6QX7YCmfjdJ

--HG--
extra : rebase_source : 0610d67f376c462b0f103a6510f21459fc39f940
2017-09-06 22:52:46 -04:00
Andrew Halberstadt f79b06a32a Bug 1339178 - Use pytest to run python-tests, r=davehunt
This switches most tests over to use pytest as the runner instead of unittest (taking
advantage of the fact that pytest can run unittest based tests).

There were a couple tests that had failures when swithing to pytest:
config/tests/unit-expandlibs.py
xpcom/idl-parser/xpidl/runtests.py

For these tests, I added a runwith='unittest' argument so that they still run the
same way as before. Once we fix them to use pytest, the unittest logic in mozunit.py
can be deleted.

MozReview-Commit-ID: Gcsz6z8MeOi

--HG--
extra : rebase_source : 3c762422ce0af54cbbe7d9fc20085a2d1ebe7057
2017-08-29 14:50:33 -04:00
Andrew Halberstadt 2255a9eed7 Bug 1395126 - Support cascading configuration for flake8, r=bc
This allows .flake8 files to override one another, and fixes a pretty bad known
bug with our flake8 implementation. For example, say we have a .flake8 file at:
/foo/.flake8

Before this patch, if we ran |mach lint foo/bar|, the configuration defined in
that .flake8 file wouldn't get picked up. It would only work if running the
specific directory that contains it, e.g |mach lint foo|.

This change additionally allows multiple .flake8 files to be used. So if
there's one defined at both:
/.flake8
/foo/.flake8

Then running |mach lint foo/bar| will first apply the root .flake8, then the
one under /foo (overriding earlier configuration).

This bug still doesn't make flake8 configuration perfect though. Any directory
containing a .flake8 file still needs to be explicitly listed in the "include"
section of /tools/lint/flake8.yml. Otherwise in the example above, if running
|mach lint /|, it wouldn't be able to find /foo/.flake8. This is a hard problem
and is likely best solved by fixing flake8's upstream configuration handling.

Unfortunately this means we still can't switch from a whitelist to a blacklist.

MozReview-Commit-ID: 3DZAi1QHYYo

--HG--
extra : rebase_source : 51298c5847f6c2792581d9b312c87b70fa716ee1
2017-08-29 17:32:31 -04:00
Ted Mielczarek afbf75cb40 bug 1371992 - make mozlint's LintRoller use concurrent.futures. r=ahal
There's a persistent test failure in automation that seems to have to do
with shutting down the `multiprocessing.Manager` that's used to get a
`Queue` to submit jobs to worker processes. After toying around with fixing
that I decided it would be simpler to just use concurrent.futures here,
since we already have it in-tree and it fits the use case here better
than using raw multiprocessing.

MozReview-Commit-ID: 8DdSvs2qp0q

--HG--
extra : rebase_source : 0b1cbb96bd3016778e4974a311722a8882f87216
2017-08-29 10:30:12 -04:00
Tom Prince c29574c405 Bug 1390699 - Follow-up: Use find_executable() to locate echo. r=ahal
MozReview-Commit-ID: QY8dajeXs0
2017-08-17 00:19:12 -06:00
Tom Prince fdff416cd5 Bug 1390699 - Don't try to test mozlint's --edit if `echo` isn't available; r=ahal
MozReview-Commit-ID: IMS91WthZtq

--HG--
extra : rebase_source : 38e33b07fb7fdc4916fbb1469a22f19d871298aa
2017-08-15 16:45:00 -06:00
Andrew Halberstadt 411adfbc83 Bug 1387555 - [mozlint] Make use of quickfix when using --edit with vim/nvim, r=dylan
MozReview-Commit-ID: BlJbWVv1CeO

--HG--
extra : rebase_source : 005f72d9f74ccd86ae73a298acde6b0b3fd6a550
2017-08-09 10:03:02 -04:00
Andrew Halberstadt 81abb80919 Bug 1387555 - [mozlint] After using --edit, run lint check again, r=dylan
MozReview-Commit-ID: BlJbWVv1CeO

--HG--
extra : rebase_source : 18720a32c8ea562af1602ce4c32e46173c0ca4fb
2017-08-09 10:02:23 -04:00
Andrew Halberstadt 10f8b7e161 Bug 1379151 - Add --fix and --edit to mozlint, r=standard8
While --fix previously worked with eslint, it is now more official (will show
up in the |mach lint --help|).  ESlint is still the only thing that implements
it, but we can implement it for flake8 using the `autopep8` module.

--edit is a new concept that will open an editor for each failing file to let
you fix the errors manually.  For now it is very naive (just opens the file),
and is only really useful if you have an editor integration for the linter(s).
But in the future I'd like to have editor-specific implementations for this.
For example, with vim, we can use -q to pass in an error file that will start
the editor pre-populated with a list of all errors that can then be easily
jumped to. Other editors may just open up to the line containing the error.

--fix and --edit can be used in conjunction with one another. Doing that means
only errors that can't be fixed automatically will show up in your editor.


MozReview-Commit-ID: 5JJJhMIrMIB

--HG--
extra : rebase_source : 2f78a77a91133d7fcc5620ecd5caa500decbce1b
2017-08-10 09:21:17 -04:00
Sebastian Hengst aedd2b0e08 Backed out changeset d24be9fbef98 (bug 1379151) for breaking Windows builds. r=backout on a CLOSED TREE 2017-08-10 16:55:11 +02:00
Andrew Halberstadt eb06963c9e Bug 1379151 - Add --fix and --edit to mozlint, r=standard8
While --fix previously worked with eslint, it is now more official (will show
up in the |mach lint --help|).  ESlint is still the only thing that implements
it, but we can implement it for flake8 using the `autopep8` module.

--edit is a new concept that will open an editor for each failing file to let
you fix the errors manually.  For now it is very naive (just opens the file),
and is only really useful if you have an editor integration for the linter(s).
But in the future I'd like to have editor-specific implementations for this.
For example, with vim, we can use -q to pass in an error file that will start
the editor pre-populated with a list of all errors that can then be easily
jumped to. Other editors may just open up to the line containing the error.

--fix and --edit can be used in conjunction with one another. Doing that means
only errors that can't be fixed automatically will show up in your editor.


MozReview-Commit-ID: 5JJJhMIrMIB

--HG--
extra : rebase_source : 889c825dcde5e950e35bc64539b299988f82ac68
2017-08-10 09:21:17 -04:00
Justin Wood f66ff278ca Bug 1387862 - Lint python/mozlint yaml files. r=ahal
We should have CI Lint YAML files in the tree.

MozReview-Commit-ID: IMOKGhxKFJW

--HG--
extra : rebase_source : 8c695dcb3a81b6683bc76ba663feeacb3126691e
2017-08-06 13:43:04 -04:00
Sebastian Hengst ca07202265 Backed out changeset 103073e92350 (bug 1387862) 2017-08-10 14:29:48 +02:00
Justin Wood 582030b224 Bug 1387862 - Lint python/mozlint yaml files. r=ahal
We should have CI Lint YAML files in the tree.

MozReview-Commit-ID: IMOKGhxKFJW

--HG--
extra : rebase_source : a2e47f5c5517e999fe21bac4f86491611bd0cece
2017-08-06 13:43:04 -04:00
Ryan VanderMeulen 24396489e3 Backed out 7 changesets (bug 1387862) for yaml linting failures.
Backed out changeset 63f87f6db7d6 (bug 1387862)
Backed out changeset a85b7e7d9f24 (bug 1387862)
Backed out changeset 3713ea9672e8 (bug 1387862)
Backed out changeset 22c1094e387f (bug 1387862)
Backed out changeset e0bfb35b0eec (bug 1387862)
Backed out changeset 5bb5dc7655ec (bug 1387862)
Backed out changeset cc4c01794114 (bug 1387862)
2017-08-09 21:31:30 -04:00
Justin Wood 906fda8397 Bug 1387862 - Lint python/mozlint yaml files. r=ahal
We should have CI Lint YAML files in the tree.

MozReview-Commit-ID: IMOKGhxKFJW

--HG--
extra : rebase_source : 0ef71f24141a531833d48f2a305183dd808f00f5
2017-08-06 13:43:04 -04:00
Wes Kocher 223f322400 Merge m-c to autoland a=merge
MozReview-Commit-ID: 2qLtb79Nlhs
2017-08-08 15:26:30 -07:00
Dyex db4f4de93a Bug 1369710 - [mozlint] Ensure that a valid path is entered. r=ahal
MozReview-Commit-ID: 6HOE9hmOkCx

--HG--
extra : rebase_source : 584a4391b832e794dc084254e9203b3a65c89419
2017-08-08 19:20:31 +05:30
Justin Wood 4a6556fa3c Bug 1387830 - Make ./mach lint able to output available linters. r=ahal
MozReview-Commit-ID: 2hRtfzohwTR

--HG--
extra : rebase_source : 18275b1015f19b98461ebb849555a02fed335462
2017-08-06 09:41:05 -04:00
Wes Kocher 0854f5772d Backed out changeset 393b0727cba4 (bug 1387830) for flake8 failures a=backout
MozReview-Commit-ID: IOTenjMeVmt
2017-08-08 09:19:39 -07:00
Justin Wood 5381aa19f5 Bug 1387830 - Make ./mach lint able to output available linters. r=ahal
MozReview-Commit-ID: 2hRtfzohwTR

--HG--
extra : rebase_source : 7658b23e5456377bca9d20b5be93b40ca3c65ecc
2017-08-06 09:41:05 -04:00
Andrew Halberstadt efd355d4d2 Bug 1306122 - [mozlint] Create a compact formatter that mimics the eslint 'compact' format, r=armenzg
MozReview-Commit-ID: 5JJJhMIrMIB

--HG--
extra : rebase_source : 2c8388e20005f7cc67c31fd631e96def452e5416
2017-08-04 10:53:43 -04:00
Andrew Halberstadt 6e2032d8c4 Bug 1361972 - [mozlint] Add ability to only lint staged changes to --workdir with git r=standard8
MozReview-Commit-ID: DUxCKN2fiag

--HG--
extra : rebase_source : 11bb7808c7d0f1c02eaea93dc61723d17e921eeb
2017-06-30 18:29:31 -07:00
Andrew Halberstadt 96fe8190c2 Bug 1375834 - [mozlint] Stop printing vcs command used if command returns non-zero, r=bc
MozReview-Commit-ID: HfG5CqQDIdr

--HG--
extra : rebase_source : 062d201073a13a906e156cee6592ef5251ab39d6
2017-06-23 08:55:04 -04:00
Andrew Halberstadt af86c88b2f Bug 1369787 - [mozlint] Add a test for vcs operations, r=bc
Create a test for version control related functionality.

MozReview-Commit-ID: GXd27O69GNg

--HG--
extra : rebase_source : 56ce4a38b591fd62f05fbaed0ff05d56ec127422
2017-06-08 14:28:37 -04:00
Andrew Halberstadt 4761330083 Bug 1369787 - [mozlint] Fix up version control commands, r=standard8
The underlying commands to mozlint's vcs.py had a few problems. This:

    1. Ensures deleted files aren't considered in both hg and git
    2. Automatically determines the default remote repo and branch when using --outgoing with git
    3. Excludes metadata from output of the git command used with --outgoing
    4. Adds --cached to the git command for --workdir, which lints all staged files

MozReview-Commit-ID: EBtM3MCiCDs

--HG--
extra : rebase_source : 982b24acd81c86e383b28b8a71bcd51a041e60f4
2017-06-08 23:31:16 -04:00
Andrew Halberstadt 5fde640164 Bug 1369787 - [mozlint] Refactor vcs.py into separate classes for hg and git, r=bc
The old method had a bunch of 'if vcs == git' statements and the like. This
made it hard to follow code paths for a given vcs system. I think this refactor
makes things cleaner to read. It shouldn't be changing any functionality.

MozReview-Commit-ID: EBtM3MCiCDs

--HG--
extra : rebase_source : 9c0fda3f4f824351bae852af25bcd2240b1b1024
2017-06-08 23:31:02 -04:00
Andrew Halberstadt 91efad9d6d Bug 1369787 - [mozlint] Remove 'rev' option from |mach lint|, r=bc
The rev option is inherently broken. It does let you lint files touched by any
revision, but it doesn't update those files to that revision first.  Instead,
they get linted at whatever the working directory is and their results are
bogus. Even if we did some magic to update the files to the proper revision
with in-memory version control magic, the config files would still be out of
date.

Plus, the new --outgoing option does pretty much the only thing --rev was good
for. Rather than cause confusion, I think it's better to just remove the
option.

MozReview-Commit-ID: 2y2UnfIkvsR

--HG--
extra : rebase_source : 9b5c142270c98905d71ebb89d1620e91914c0b47
2017-06-08 22:54:00 -04:00
Andrew Halberstadt 4b93103061 Bug 1375166 - [mozlint] Don't require leading '.' in extensions, r=standard8
This was a regression from bug 1288432. The 'extensions' config in mozlint required a
leading period, but eslint requires them without the period (and this got copied over
to the linter definition). The result was mozlint filtering out any files (not dirs)
that were passed in.

This just modifies mozlint to strip out the period so both are acceptable.

MozReview-Commit-ID: CbNynYzrbGz

--HG--
extra : rebase_source : 51c740cb1d2febaee3ae46784f83381cda5e5eaa
2017-06-21 13:22:34 -04:00
Andrew Halberstadt 9a082a086c Bug 1288432 - [mozlint] Use yaml for lint definitions and separate implementation of external linters, r=bc
Rather than using .lint.py files that contain a LINTER object, linter definitions are now in
standalone .yml files. In the case of external linters that need to run python code, the payload
is now of the form:
<module path>:<object path>

The <module path> is the import path to the module, and <object path> is the callable object to
use within that module. It is up to the consumer of mozlint to ensure the <module path> lives on
sys.path. For example, if an external lint's function lives in package 'foo', file 'bar.py' and
function 'lint', the payload would read:
foo.bar:lint

This mechanism was borrowed from taskcluster.

MozReview-Commit-ID: AIsfbVmozy4

--HG--
rename : python/mozlint/test/linters/badreturncode.lint.py => python/mozlint/test/linters/badreturncode.yml
rename : python/mozlint/test/linters/explicit_path.lint.py => python/mozlint/test/linters/explicit_path.yml
rename : python/mozlint/test/linters/external.lint.py => python/mozlint/test/linters/external.yml
rename : python/mozlint/test/linters/invalid_exclude.lint.py => python/mozlint/test/linters/invalid_exclude.yml
rename : python/mozlint/test/linters/invalid_extension.lnt => python/mozlint/test/linters/invalid_extension.ym
rename : python/mozlint/test/linters/invalid_include.lint.py => python/mozlint/test/linters/invalid_include.yml
rename : python/mozlint/test/linters/invalid_type.lint.py => python/mozlint/test/linters/invalid_type.yml
rename : python/mozlint/test/linters/missing_attrs.lint.py => python/mozlint/test/linters/missing_attrs.yml
rename : python/mozlint/test/linters/missing_definition.lint.py => python/mozlint/test/linters/missing_definition.yml
rename : python/mozlint/test/linters/raises.lint.py => python/mozlint/test/linters/raises.yml
rename : python/mozlint/test/linters/regex.lint.py => python/mozlint/test/linters/regex.yml
rename : python/mozlint/test/linters/string.lint.py => python/mozlint/test/linters/string.yml
rename : python/mozlint/test/linters/structured.lint.py => python/mozlint/test/linters/structured.yml
extra : rebase_source : bda3926712234123355c5af71c6453ce869b19fc
2017-06-02 09:49:26 -04:00
Mark Banner 021f1ff90a Bug 1358540 - Change the *.lint files to be *.lint.py to better support editor integration & flake8 linting. r=smacleod
MozReview-Commit-ID: 4KK2GZK7xul

--HG--
rename : python/mozlint/test/linters/badreturncode.lint => python/mozlint/test/linters/badreturncode.lint.py
rename : python/mozlint/test/linters/explicit_path.lint => python/mozlint/test/linters/explicit_path.lint.py
rename : python/mozlint/test/linters/external.lint => python/mozlint/test/linters/external.lint.py
rename : python/mozlint/test/linters/invalid_exclude.lint => python/mozlint/test/linters/invalid_exclude.lint.py
rename : python/mozlint/test/linters/invalid_include.lint => python/mozlint/test/linters/invalid_include.lint.py
rename : python/mozlint/test/linters/invalid_type.lint => python/mozlint/test/linters/invalid_type.lint.py
rename : python/mozlint/test/linters/missing_attrs.lint => python/mozlint/test/linters/missing_attrs.lint.py
rename : python/mozlint/test/linters/missing_definition.lint => python/mozlint/test/linters/missing_definition.lint.py
rename : python/mozlint/test/linters/raises.lint => python/mozlint/test/linters/raises.lint.py
rename : python/mozlint/test/linters/regex.lint => python/mozlint/test/linters/regex.lint.py
rename : python/mozlint/test/linters/string.lint => python/mozlint/test/linters/string.lint.py
rename : python/mozlint/test/linters/structured.lint => python/mozlint/test/linters/structured.lint.py
rename : tools/lint/eslint.lint => tools/lint/eslint.lint.py
rename : tools/lint/flake8.lint => tools/lint/flake8.lint.py
rename : tools/lint/wpt.lint => tools/lint/wpt.lint.py
rename : tools/lint/wpt_manifest.lint => tools/lint/wpt_manifest.lint.py
extra : rebase_source : 97de88e0328e91347192672b848b286df1904ade
2017-04-21 17:31:15 +01:00
Carsten "Tomcat" Book 1dd975c180 Backed out changeset eec770d062b4 (bug 1358540) for bustage
--HG--
rename : tools/lint/eslint.lint.py => tools/lint/eslint.lint
rename : tools/lint/flake8.lint.py => tools/lint/flake8.lint
rename : tools/lint/wpt.lint.py => tools/lint/wpt.lint
rename : tools/lint/wpt_manifest.lint.py => tools/lint/wpt_manifest.lint
2017-04-25 10:15:07 +02:00
Mark Banner 8e42062ea3 Bug 1358540 - Change the *.lint files to be *.lint.py to better support editor integration & flake8 linting. r=smacleod
MozReview-Commit-ID: 4KK2GZK7xul

--HG--
rename : tools/lint/eslint.lint => tools/lint/eslint.lint.py
rename : tools/lint/flake8.lint => tools/lint/flake8.lint.py
rename : tools/lint/wpt.lint => tools/lint/wpt.lint.py
rename : tools/lint/wpt_manifest.lint => tools/lint/wpt_manifest.lint.py
extra : rebase_source : 66e4b4f95109c2645d0fb7be273de343408b9406
2017-04-21 17:31:15 +01:00
Andrew Halberstadt 8d7db75ee7 Bug 1340162 - Add task for running mozlint unitttests on Linux (and pull old tests out of make check), r=smacleod
MozReview-Commit-ID: 3XCWMJtQMvZ

--HG--
extra : rebase_source : 290c14cda5b7abf6aea43315fc91c0ac8fc869f0
2017-02-02 11:08:41 -05:00
Mike Hommey dc5d1e6adf Bug 1335309 - Change the default for find_executables to False. r=mshal
Back when the class was written, for the packaging code, it made sense
that the default was True. But now that it's used all over the place,
and that the vast majority of uses are with find_executables=False, it
makes more sense for that to be the default.

--HG--
extra : rebase_source : ff813735fc0d53093f348f20eb77ee03e9b09d4e
2017-01-31 14:06:15 +09:00
Francesco Pischedda 8a2906dd4f Bug 1313194 - Add --outgoing argument to mozlint. r=ahal
Was: Replace mozlint's --rev with a --outgoing argument

MozReview-Commit-ID: 5YcRWbc1dR4

--HG--
extra : rebase_source : e13fca582d37fb40ba9d7782413190da4c1fa8b3
2017-01-26 17:50:34 +01:00
Andrew Halberstadt b994a9eca5 Bug 1318597 - Explicitly call manager.shutdown() after running mozlint, r=mshal
This is an educated guess in an attempt to solve a hard to reproduce intermittent. This
also makes sure the original SIGINT handler is restored after running the linters.

MozReview-Commit-ID: HRo0GRlSN5L

--HG--
extra : rebase_source : 7b8bf4b5818d741b5ab6a38c8c56f4eb53a801c8
2017-01-23 17:35:03 -05:00
Andrew Halberstadt 6adcf5b456 Bug 1317970 - Use manifestparser manifests for python unit tests, r=chmanchester
This deprecates PYTHON_UNIT_TESTS and replaces it with PYTHON_UNITTEST_MANIFESTS.
In the build system, this means python unittests will be treated the same as all
other test suites that use manifestparser. New manifests called 'python.ini' have
been created for all test directories containing python unittests.

MozReview-Commit-ID: IBHG7Thif2D

--HG--
extra : rebase_source : 11a92a2bc544d067946bbd774975140147458caa
2016-11-16 09:59:22 -05:00
Andrew Halberstadt 4524d8939a Bug 1318488 - [mozlint] Use sys.stdout.encoding when printing formatter's output, r=smacleod
This fixes a UnicodeDecodeError when sys.stdout's encoding can't handle unicode.

MozReview-Commit-ID: 3INna8MRje5

--HG--
extra : rebase_source : bb8f3b5cbdbb6601b6946d5fbb7396848f22b5c6
2016-11-17 17:36:34 -05:00
Andrew Halberstadt 7c26ce739c Bug 1316925 - Keep track of failed linters in stylish formatter summary, r=jgraham
This replaces the "return_code" property on the LintRoller object with a list of "failed"
linters. This is a bit more useful as it lets us know exactly which linters had a problem
(whereas previously we just knew *something* went wrong). This patch pushes determining
the return code back into cli.py, which I think is fine.

In addition, we now pass the list of failed linters into the formatter. This allows us to
clarify exactly how many linters hit a failure which is a lot better than a seemingly
"successful" summary message.

Finally I also removed the "no files to lint" message because I've seen several people
confuse it for an error. I'll probably add it back as a debug log message when we switch
to using mozlog for output.

MozReview-Commit-ID: 4wyCeOZdOf8

--HG--
extra : rebase_source : 23810a8ab8dd9cbbad6b9e965ccff7214f947fbc
2016-11-14 11:56:46 -05:00
Andrew Halberstadt 5b45c5a120 Bug 1315805 - [mozlint] Make sure multiprocess' stdout gets flushed in lint tasks, r=dustin
MozReview-Commit-ID: FhOI92JySXg

--HG--
extra : rebase_source : f3de6198a2616315358e3e3be1441df0713b442f
2016-11-08 13:33:13 -05:00
Andrew Halberstadt dde3965629 Bug 1309963 - Make sure return codes are passed from .lint files up to the cli, r=jgraham
If a linter returns a status code instead of a list of results, mozlint will throw away
that status code and move on to the next linter. We should make sure that status code
bubbles all the way up to the cli.

MozReview-Commit-ID: 9fXpmtlUUT1

--HG--
extra : rebase_source : 84c80d1b24d7233d0c4c6e38b23e775110776d1f
2016-10-13 16:23:13 -04:00
James Graham bba975bca2 Bug 1302796 - Add integration between structured logging and lints, r=ahal
MozReview-Commit-ID: K3tu0Zdg5go

--HG--
extra : rebase_source : 2b78059908c3a297b8ce7d134af20733fc082df5
2016-09-13 14:18:41 +01:00
Andrew Halberstadt 98a0fd6336 Bug 1302172 - [mozlint] Convert unittest tests to use pytest instead, r=maja_zf
MozReview-Commit-ID: D4bN62QbkKm

--HG--
extra : rebase_source : 22ffe05ca78095fc469520b9f85d93585251cd1c
2016-09-09 16:20:09 -04:00
Andrew Halberstadt 0f35e014fa Bug 1300779 - [mozlint] Redirect stderr when detecting version control root, r=smacleod
This avoids printing harmless (but confusing) errors to the log. For instance, git users
will see a '.hg not found!' error in the output even though not finding an hg repo is
expected in that case.

MozReview-Commit-ID: DBPOabcV7PA

--HG--
extra : rebase_source : a40de185cd4bf0c58917065d1331e0e663b92b94
2016-09-06 10:36:45 -04:00
Andrew Halberstadt 1ec1f86a90 Bug 1299618 - [mozlint] Encode results to utf-8 before printing to stdout, r=smacleod
MozReview-Commit-ID: 93RAqIRKAHc

--HG--
extra : rebase_source : d1973e52e72baab13ba9c8f1b07df5a7ddda7d49
2016-08-31 17:18:52 -04:00
Mike Hommey 411867285a Bug 1299661 - Use hg --template flag instead of -T. r=gps
Versions of mercurial older than 3.0 don't support the -T shorthand for
the --template option. While most people should be using something
newer, it can still happen that some run an older version, and it's
still trivial to support them by using the long option.

--HG--
extra : rebase_source : 1507aea436779495045df48b044a58f4af1382be
2016-09-01 07:43:38 +09:00
Andrew Halberstadt c88eb083d8 Bug 1258341 - [mozlint] Add ability to forward miscellaneous command line arguments to the underlying linter, r=smacleod
This is mostly to maintain backwards compatibility with the |mach eslint| command. But it's also going to be used in
automation. The 'mozlint-eslint' task will need to pass in --quiet to eslint.

Maybe in the future we should remove this ability and only allow well-defined arguments in mozlint. But for now it's
convenient and allows us to land the eslint->mozlint patch series quicker.

MozReview-Commit-ID: KYhC6SEySC3

--HG--
extra : rebase_source : ad7f0cdeb57ed8a99becb90d761d73ef45ce853a
2016-08-25 09:52:50 -04:00
Andrew Halberstadt ae1bcceeda Bug 1297699 - Mozlint tests should run with test directory as the "project root", r=smacleod
MozReview-Commit-ID: 9sbhlc11YF5

--HG--
extra : rebase_source : dee2bc4820374ca833f8351de72743f8d5a95aec
2016-08-24 09:55:04 -04:00
Andrew Halberstadt ed73940aa9 Bug 1288425 - Make sure we skip invalid extensions when linting with --rev or --workdir, r=smacleod
Some linters, such as flake8, will lint invalid file extensions if you explicitly pass them in. E.g,
|flake8 foobar.js| will result in flake8 attempting to lint a JS file. This is a problem because passing
in files explicitly is exactly what the --rev/--workdir options do. If a developer modifies a JS file
then runs |mach lint -l flake8 -w|, that JS file will get linted.

To prevent this, mozlint needs to handle file extensions instead of relying on the underlying linter to
do it. This patch adds an "extensions" config option to the LINTER dict, and will filter these files out
as part of the 'filterpaths' steps.

MozReview-Commit-ID: KYhC6SEySC3

--HG--
extra : rebase_source : 6fea2942b2db1bea7deca1d6738546362b6ebd65
2016-08-09 16:24:04 -04:00
Andrew Halberstadt 797aa910b7 Bug 1294802 - Improve color handling in mozlint stylish formatter, r=jgraham
This will make specifying terminal colours cleaner and provide fallbacks for terminals
with only 8 colours. It also tweaks the 'grey' color to match an update to the eslint
stylish formatter.

MozReview-Commit-ID: KYhC6SEySC3

--HG--
extra : rebase_source : 8d3242edc54bf6505aa2e60cdb08cb0dd9743dd3
2016-07-19 11:54:04 -04:00
Andrew Halberstadt 8c3bcff5a6 Bug 1289805 - Properly normalize all paths so they are absolute, r=smacleod
It's important to use absolute paths so lint errors for the same files don't show up in two
different places. For example, eslint will absolutize a relative path, whereas flake8 will
not.

This patch also makes sure all include/exclude paths are joined to the mozlint 'root' that
was previously calculated from vcs.

MozReview-Commit-ID: KYhC6SEySC3

--HG--
extra : rebase_source : 9b88ee70a997d3f7b6cc0eb849e47931d5726f20
2016-07-19 13:09:38 -04:00
Andrew Halberstadt 560ce6a254 Bug 1289805 - Resolve vcs arguments directly in LintRoller class, r=smacleod
Previously, vcs related stuff was resolved in the cli.py module. But it's possible
for consumers to bypass the cli and instantiate a LintRoller directly. In fact this
is what the mozlint tests do.

Now that we always try to find the vcs root, calling into vcs is no longer optional.
This patch moves the VCSFiles class to a new vcs.py module and makes LintRoller
responsible for instantiating it instead of cli.py.

MozReview-Commit-ID: 5yA3gDZ1UGM

--HG--
extra : rebase_source : 96adc0ca01e8220b0432b64c96c478fc526db756
extra : source : 709c1e521ad3eb466c3434000aed7f71ebf37365
2016-08-10 10:21:43 -04:00
Andrew Halberstadt ce3f82ffbe Bug 1289805 - Ensure the VCSFiles class returns paths that have been absolutized to the repository root, r=smacleod
This makes sure we return absolute paths from the VCSFiles class. This is done so we don't accidentally normalize
the relative paths onto $CWD in the event the relative path is valid in both places.

MozReview-Commit-ID: 2QIuR2YqFos

--HG--
extra : rebase_source : 2e82b4a5cf9a18856bbcb8d04fdc916ff7e72deb
2016-08-09 14:49:41 -04:00
Andrew Halberstadt 64bfc88f52 Bug 1289805 - Refactor filterpaths to accept all lintargs, r=smacleod
This makes it easier to pass configuration to the filterpaths method. In
the future, 'root' and 'extensions' will both be used here.

MozReview-Commit-ID: KYhC6SEySC3

--HG--
extra : rebase_source : 19ddf511ebd24a1a6c21f7c06b67ab64e78e7903
2016-08-09 15:29:49 -04:00
Iris Hsiao bb2d612158 Backed out changeset 2141360b4137 (bug 1288425) for build bustage 2016-07-27 10:59:10 +08:00
Andrew Halberstadt 18f9c226a5 Bug 1288425 - Make sure we skip invalid extensions when linting with --rev or --workdir, r=smacleod
Some linters, such as flake8, will lint invalid file extensions if you explicitly pass them in. E.g,
|flake8 foobar.js| will result in flake8 attempting to lint a JS file. This is a problem because passing
in files explicitly is exactly what the --rev/--workdir options do. If a developer modifies a JS file
then runs |mach lint -l flake8 -w|, that JS file will get linted.

To prevent this, mozlint needs to handle file extensions instead of relying on the underlying linter to
do it. This patch adds an "extensions" config option to the LINTER dict, and will filter these files out
as part of the 'filterpaths' steps.

MozReview-Commit-ID: KYhC6SEySC3

--HG--
extra : rebase_source : 46807a4913660f33e691b864c2c59c448902dfa5
2016-07-19 13:50:25 -04:00
Tom Tromey 5538d692d3 Bug 1286877 - do not set c-basic-offset for python-mode; r=gps
This removes the unnecessary setting of c-basic-offset from all
python-mode files.

This was automatically generated using

    perl -pi -e 's/; *c-basic-offset: *[0-9]+//'

... on the affected files.

The bulk of these files are moz.build files but there a few others as
well.

MozReview-Commit-ID: 2pPf3DEiZqx

--HG--
extra : rebase_source : 0a7dcac80b924174a2c429b093791148ea6ac204
2016-07-14 10:16:42 -06:00
Andrew Halberstadt d5b1ff9f2e Bug 1287834 - [mozlint] Fix regression preventing specific linters from being run, r=smacleod
MozReview-Commit-ID: KYhC6SEySC3

--HG--
extra : rebase_source : bce3e321f407b89d65f1491be74384c0ae171431
2016-07-19 11:50:33 -04:00
Andrew Halberstadt 3c8be23f37 Bug 1281899 - [mozlint] Add ability to lint files touched by revisions and/or the working directory, r=smacleod
This adds two parameters, --rev and --workdir. Each works both with mercurial and git (though the syntax for
specifying revisions is different between them). The value is simply forwarded to either |hg log| or |git diff|
so syntax like |mach lint -r .~4::.| or |mach lint -r "HEAD~4 HEAD"| will work as expected.

MozReview-Commit-ID: aOGp2Yrncs

--HG--
extra : rebase_source : d2cb834d4cc1a083171a3551af4e72c8a7d14021
2016-06-24 14:09:58 -04:00
Andrew Halberstadt 482f8a956c Bug 1281899 - [mozlint] Create cli module and move logic from tools/lint/mach_commands.py there, r=smacleod
There is currently no built-in user interface to mozlint. The only existing interface is the
external cli provided by |mach lint|. However, in the future mozlint may need to be used in a
context where mach isn't readily available (i.e version-control-tools). This patch basically
just moves the cli logic out of mach_commands.py, and into mozlint core. That way it can be
re-used in other places without needing to be re-implemented.

The |mach lint setup| subcommand was removed because apparently subcommands don't work with
the parser attribute. Nothing was using it yet anyway, so I removed it for now. It may get
re-added in some form in the future.

MozReview-Commit-ID: aOGp2Yrncs

--HG--
extra : rebase_source : 8f7530de96e5c131d2ed5bfcdd7a159329401e5b
2016-06-24 14:06:22 -04:00
James Graham 3ca7bb37ed Bug 1279811 - Make output of column number optional in mozlint, r=ahal
MozReview-Commit-ID: 7eGTJKPMMAc
2016-06-23 10:36:47 +01:00
Andrew Halberstadt 13a551f7f9 Bug 1277641 - [mozlint] Add --no-filter option to make experimenting locally easier, r=smacleod
This makes it easier to lint a path that otherwise wouldn't have been linted due to the include/exclude
directives. Now, you can pass in -n/--no-filter instead of needing to modify the linter configuration file.

MozReview-Commit-ID: GMJuE2C1NyY

--HG--
extra : rebase_source : 03627e930f76903ad629cb01b58c4ae7372e4bb1
2016-06-02 15:14:07 -04:00
Andrew Halberstadt 031698ae90 Bug 1273634 - [mozlint] Add a treeherder formatter, r=jgraham
This is a really simple and ugly formatter that is compatible with
treeherder's error highlighting mechanism. It is designed to be identical
to the current eslint output on treeherder:
https://dxr.mozilla.org/mozilla-central/rev/4d63dde701b47b8661ab7990f197b6b60e543839/tools/lint/eslint-formatter.js

Eventually eslint will also use this and we can remove that file. Once
bug 1276486 is fixed, we can make this look a little nicer. But for now
it gets the job done.

MozReview-Commit-ID: CwfWPcwWFxF

--HG--
extra : transplant_source : %F3PJ%CB%27%A5%82U%D2%CF%B3%9E%A7%9F%0F%A4%F4%E9%5D%BB
2016-05-28 23:38:30 -04:00
Carsten "Tomcat" Book c30c6ba10e Backed out changeset c48f3b04c9de (bug 1273634) 2016-06-02 15:05:01 +02:00
Andrew Halberstadt df97806e5f Bug 1273634 - [mozlint] Add a treeherder formatter, r=jgraham
This is a really simple and ugly formatter that is compatible with
treeherder's error highlighting mechanism. It is designed to be identical
to the current eslint output on treeherder:
https://dxr.mozilla.org/mozilla-central/rev/4d63dde701b47b8661ab7990f197b6b60e543839/tools/lint/eslint-formatter.js

Eventually eslint will also use this and we can remove that file. Once
bug 1276486 is fixed, we can make this look a little nicer. But for now
it gets the job done.

MozReview-Commit-ID: CwfWPcwWFxF

--HG--
extra : rebase_source : 8dd39aefec1064e0836c847c6d223db43df4755b
2016-05-28 23:38:30 -04:00
Andrew Halberstadt 464b3a9f2e Bug 1273556 - [mozlint] Better SIGINT handling, return partial results on Ctrl-C, r=jgraham
Currently a bug in python (https://bugs.python.org/issue8296) is preventing a KeyboardInterrupt from
reaching the parent process, meaning we can't kill the process with SIGINT. There is a workaround to
this bug, but instead I decided to ignore SIGINT in the parent process completely. Now, each child
process is responsible for handling SIGINT on its own. Since child processes should all shutdown
relatively quickly anyway, this effectively also ends the parent process.

The benefit of doing it this way is that each child process can return the results they have collected
to date. So when a developer hits Ctrl-C, they'll still see some (but not all) formatted lint output.
The downside is that a poorly implemented external linter could block the parent process from exiting
quickly, but if this happens we should just fix the linter.

MozReview-Commit-ID: 2tRJgtmoPYP

--HG--
extra : rebase_source : e72282bcc29c9b22690168dc973a817236cdf8ca
2016-05-17 12:24:42 -04:00
Andrew Halberstadt b1445349ea Bug 1270506 - [mozlint] Add python flake8 linter, r=smacleod
For now, only the following two directories will be linted:
python/mozlint
tools/lint

New directories can be added by adding them to the 'include'
directive in tools/lint/flake8.lint. They all default to the
configuration specified in topsrcdir/.flake8. Subdirectories
can override this configuration by creating their own .flake8
file.

MozReview-Commit-ID: Eag48Lnkp3l

--HG--
extra : rebase_source : 6d98c9fef80055a48cc2622848aa04c3045b747e
2016-05-05 17:21:12 -04:00
Andrew Halberstadt 5f616862e3 Bug 1270506 - [mozlint] Refactor the include/exclude path filtering algorithm, r=smacleod
The current algorithm for filtering down tests is too naive. For example, given the following
directory structured:

parent
  - foo
    - bar
    - baz

And the following include/exclude directives:
include = ['foo']
exclude = ['foo/bar']

Then running ./mach lint parent and ./mach lint foo/baz should both lint all files in baz but
no files in bar. This provides a nice way to include/exclude directories, while allowing the
underlying linters to find appropriate files to lint *within* those directories.

tl;dr - Straight file paths (no globs) will be passed straight to the underlying linter as is.
While paths with globs will first be resolved to a list of matching file paths.

MozReview-Commit-ID: Eag48Lnkp3l

--HG--
extra : rebase_source : 18ce2231bc4198b1e811d39f9803f0d5e03d982e
2016-05-05 17:20:33 -04:00
Andrew Halberstadt bb96d51342 Bug 1230962 - Add python/mozlint for running several linters at once, r=smacleod
Mozlint provides two main benefits:
1. A common system for defining lints across multiple languages
2. A common interface and result format for running them

This commit only adds the core library, it does not add any consumers of mozlint just yet.

MozReview-Commit-ID: CSQzq5del5k

--HG--
extra : rebase_source : b520b96177281a1b1770edf53a01cbc2196f494f
2016-03-16 14:55:21 -04:00