Граф коммитов

43 Коммитов

Автор SHA1 Сообщение Дата
Yoann Schneider 05a7deae77 N818: Exception name should be named with an Error suffix 2024-03-05 16:17:10 +01:00
Yoann Schneider cfb19a5ef8
Bug 1823654 - Mostly use double quotes (#7900)
* Enable quote style verifications from Black

* Actually exclude migrations folders
2024-01-31 16:59:20 +01:00
Cameron Dawson 1250108abe
Upgrade pip dev packages (#6742)
* Upgrade pip dev packages
* Fix formatting with black
2020-08-31 11:01:43 -07:00
Sarah Clements 507881a7b8
Remove step parser v3 (#6634)
* remove StepParser and switch to ErrorParser
* remove writes to TextLogStep from artifact.py
* remove buildbot ref in builders
* replace TextLogStep model in DetailsPanel, SimilarJobsTab and logviewer App
* cleanup DetailsPanel
* remove old log parsing tests and update others
* add logging to error_summary.py
* add parse max error lines limit to ErrorParser
* fix in similar jobs tab for Bug 1652869
2020-07-17 10:28:01 -07:00
Armen Zambrano G 7430583ff7 Revert "Bug 1633041 - Remove step parser v2 (#6611)"
This reverts commit 1d1be59d29.
2020-07-16 09:35:52 -04:00
Sarah Clements 1d1be59d29
Bug 1633041 - Remove step parser v2 (#6611)
* remove StepParser and stop storing steps in TextLogStep
* remove more buildbot references 
* replace TextLogStep model in DetailsPanel and logviewer App
* remove old log parsing tests and update other tests
* add logging to error_summary.py
2020-07-14 10:57:09 -07:00
Sarah Clements bed22bf0da
Revert "Bug 1633041 - Remove step parser and TextLogStep table (#6546)" (#6598)
This reverts commit 7d3d8ae902.
2020-07-09 11:04:04 -07:00
Sarah Clements 7d3d8ae902
Bug 1633041 - Remove step parser and TextLogStep table (#6546)
* remove StepParser and stop storing steps
* replace TextLogStepModel in the UI with text-log-errors API
* remove old references to buildbot
* update and cleanup tests
2020-07-08 14:57:40 -07:00
Sarah Clements d598ad4544
Bug 1603249 - Remove job artifacts storage and TinderboxPrint parsing (#6349)
* remove artifact storage in the job loader
* remove BuildbotJobArtifactBuilder and TinderboxPrint parser
* update pulse-job schema and tests
2020-05-07 09:54:06 -07:00
Armen Zambrano 368c112266
Apply black formating to the whole repo (#6332)
* Automatic black fixes
* Add conflicting rule exceptions between black and flake8
2020-04-21 10:35:59 -04:00
dependabot[bot] c355983fe5 Bump isort from 4.3.10 to 4.3.12 in /requirements (#4760)
Bumps [isort](https://github.com/timothycrosley/isort) from 4.3.10 to 4.3.12.
- [Release notes](https://github.com/timothycrosley/isort/releases)
- [Changelog](https://github.com/timothycrosley/isort/blob/develop/CHANGELOG.md)
- [Commits](https://github.com/timothycrosley/isort/commits)

Signed-off-by: dependabot[bot] <support@dependabot.com>
2019-03-06 14:54:00 -08:00
Ed Morley 52d6017c5b
Bug 1295997 - Skip parsing logs whose compressed size exceeds 5MB (#4700)
Occasionally failing build/test runs can fail in such a way that results
in a significant amount of log spam and therefore log files that are
hundreds of MB in size each. This can cause log parsing backlogs,
particularly when many jobs on the same push fail in such a way.

The log parser now checks the `Content-Length` of log files prior to
streaming them, and skips the download/parse if it exceeds the set
threshold. The frontend has been adjusted to display an appropriate
message explaining why the parsed log is not available.

The threshold has been set to 5MB, since:
* the 99th percentile of download size on New Relic was ~2.8MB:
  https://insights.newrelic.com/accounts/677903/dashboards/339080
* `Content-Length` is the size of the log prior to decompression, and
  the chronic logspam cases have been known to have compression ratios
  of 20-50x, which would translate to an uncompressed size limit of
  up to 250MB (which is already much larger than buildbot's former 50MB
  uncompressed size limit).
2019-02-25 19:04:38 +00:00
Ed Morley c31e1dcc81
Bug 1453297 - Remove datadiff (#3435)
Since `pytest -vv` displays useful diffs on its own now.
2018-04-11 13:26:42 +01:00
Ed Morley aff2152a81 Bug 1165356 - Log parser: Remove tests workaround for file:// log URLs
Now that we're using requests, the log URL being used to access the file
is consistent across environments (since it doesn't reference the local
directory structure), so we don't need to exclude it from comparisons.
2017-03-22 00:50:02 +00:00
Ed Morley 6717d9f5fc Bug 1165356 - Log parser: Fetch logs using requests instead of urllib2
Doing so also fixes incorrect line numbers in logs that have Windows
line endings (bug 1328880), hence having to update the expected logview
output.

The tests have to be adjusted to use responses, since requests doesn't
support `file://` URLs.
2017-03-22 00:50:02 +00:00
William Lachance 28cae8c360 Bug 1302844 - Remove redundant information from text log artifacts (#1854)
Now that they are only an intermediary format, we can remove data
that isn't required to interpret them.
2016-09-21 10:54:53 -04:00
William Lachance b69d2e4515 Bug 1263691 - Remove talos parsing/processing code from Perfherder
It's no longer needed now that Talos only submits generic Perfherder artifacts
2016-04-13 16:27:12 -04:00
William Lachance ce4b22341d Bug 1149164 - Support ingestion of arbitrary perf data from logs
You can now specify performance data to be ingested from any job just
by adding a line containing PERFHERDER_DATA in it.
2015-10-28 14:34:01 -04:00
Ed Morley 90ba77e596 Bug 1192801 - Remove per-file MPL boilerplate since it's unnecessary
The MPL 2.0 terms state that as long as a LICENSE file is present, the
per-file header text is not required. See "Exhibit A" at the end of:
https://www.mozilla.org/MPL/2.0/
2015-08-18 23:32:11 +01:00
Ed Morley 095fc79e62 Bug 1188444 - Remove support for multiple parsers per ArtifactBuilder
Since we only have one parser per ArtifactBuilder currently, and are not
likely to ever have any more.
2015-08-18 17:21:02 +01:00
Ed Morley 79e7a4eced Bug 1188444 - Remove check_errors parser option since it's always true
Bug 1060339 made check_errors always be true, since we want to parse all
logs, not just those for failing jobs. As such, we have no use for
check_errors.
2015-08-18 17:21:01 +01:00
Ed Morley 1c00ccfcc7 Bug 1192661 - Clean up Python import order
Created using |isort -p tests -rc .| and a couple of manual tweaks.

The order is:
* futures
* std library
* third party packages
* local imports
* relative local imports
...with each group ordered with "import x" before "from x import y", and
then alphabetically.
2015-08-10 18:33:49 +01:00
Ed Morley 8637384d8c Bug 1175870 - Stop adding a "header" property to the text_log_summary
The header property is no longer used, since bug 1057341 made the log
viewer display job details from /jobs/ endpoint instead.
2015-06-18 18:21:50 +01:00
Cameron Dawson ca84e4f373 Bug 1113873 - Change name of ``Structured Log`` artifact to ``text_log_summary``
This is to remove confusion between the REAL structured log and the generated
summary artifact for the textual logs.
2015-03-11 08:49:10 -07:00
Ed Morley 9ce4733946 Bug 1078415 - Log parser: Only record the first 100 error lines per step
To avoid logs with excessive number of lines that match the error regex
from taking up too much space in the DB & also making the API response
and thus UI unwieldy, we cap the number of error lines at 100 per step
of the job. The 'errors_truncated' property can be used by the UI to
indicate that the error lines are only a subset of the total failures.
2015-02-16 21:10:57 +00:00
Ed Morley f5c0b53e0c Bug 1059814 - Whitespace pep8 fixes
Generated using:
autopep8 --in-place --recursive .

Before:
$ pep8 | wc -l
1686

After:
$ pep8 | wc -l
57

A later autopep8 run will be performed using --aggressive, which makes
non-whitespace changes too.
2015-02-15 14:52:31 +00:00
Jonathan French dbb4d11e09 Bug 1090689 - Add MPL2.0 headers to the repo 2014-11-03 13:06:03 -05:00
Ed Morley a623a1fa32 Bug 1060339 - Parse logs for error lines even if the job was successful
In an ideal world, we would never have a successful job that contained
errors which our log parser regex matched against. Unfortunately this is
not the case in practice, since:

1) There is sometimes spammy log output that isn't intended to make the
   job fail, but which false-positive matches against our log regex.
2) Sometimes the test/harness/buildbot/... is broken and the log error
   is real, and it should have caused the run to fail.

In the case of #1, hiding these errors in successful runs means that
when someone looks at the error summary for a failed run, they don't
realise that a proportion of the failures they can see are actually
present in _every_ job, not just the failed ones.

In the case of #2, hiding the errors means we don't realise <foo> is
broken and needs fixing.

As such, we now parse for error lines in all jobs, including successful.
2014-09-08 13:59:55 -07:00
Jonathan Eads 5a50d13862 removed print statements 2014-06-24 12:09:18 -07:00
Jonathan Eads 12385f143b fixed conditional talos data load 2014-06-24 12:03:55 -07:00
Jonathan Eads 1987e6fcd7 removed bad talos perf tests 2014-06-23 17:38:36 -07:00
Jonathan Eads 1f59f9cf5e fixing merge conflicts 2014-06-13 15:02:02 -07:00
Jonathan Eads 06b256733d merged with master 2014-06-13 14:10:55 -07:00
mdoglio 499477040d remove references to tinderbox print lines 2014-06-13 12:28:54 +02:00
pnispel fbd70064be updated tests for parsing TALOSDATA out of log 2014-05-28 09:10:25 -07:00
mdoglio f7893fd142 change artifactfuilder name from property to attribute 2013-11-29 14:24:50 +00:00
Cameron Dawson a4dde81541 adding unit test for new log parser check_errors flag 2013-09-16 15:36:46 -07:00
Cameron Dawson 5ac4e9da52 skip parsing for errors if the status of the job is success 2013-09-16 12:32:10 -07:00
Cameron Dawson 54e256eb67 new tests and performance optimizations 2013-09-13 15:34:37 -07:00
Cameron Dawson 2774233c83 fix collection test errors 2013-06-26 09:50:27 -07:00
mdoglio 6ed7a9b49f pep8 cleaning 2013-06-17 16:04:02 +02:00
Cameron Dawson e0701ebdfe finished coverage 2013-06-12 19:20:35 -07:00
Cameron Dawson cbc6862da1 fixed unit test files 2013-06-12 17:58:15 -07:00