Running tests with different JIT flags by default is inconsistent with
jstests.py or mach jit-test.
Differential Revision: https://phabricator.services.mozilla.com/D65325
--HG--
extra : moz-landing-system : lando
A quick and basic implementation of 'mach test-info report-diff' to provide a
high level comparison of two reports generated by 'mach test-info report'.
Mostly this is trying to give an impression of how many and which tests
have been added and deleted during the time period between the reports.
There is more information in the reports; a more thorough comparison is
possible, but seems lower-value: To be considered in a future bug.
Differential Revision: https://phabricator.services.mozilla.com/D65202
--HG--
extra : moz-landing-system : lando
Move almost all test-info code out of mach_commands.py and into its own file
and consolidate logging and imports; no functional changes.
Differential Revision: https://phabricator.services.mozilla.com/D56064
--HG--
extra : moz-landing-system : lando
This adds test run time and test run/skipped/failed counts for most test types.
There are numerous shortcomings (sometimes exceeding the ActiveData 10K record limit,
test path/name mismatches, no totals for manifest/component/all, etc) but I would
like to see this running asap as I think it provides a good foundation for further
work and discussion.
Differential Revision: https://phabricator.services.mozilla.com/D55576
--HG--
extra : moz-landing-system : lando
Switch 'mach test-info report' and 'mach test-info tests' to use the TestManifestLoader.
Differential Revision: https://phabricator.services.mozilla.com/D54053
--HG--
extra : moz-landing-system : lando
Puppeteer tests have t['manifest'] = None; allowing for this avoids downstream pain
when reporting manifests.
(Also removed a stray debugging print statement.)
Differential Revision: https://phabricator.services.mozilla.com/D53102
--HG--
extra : moz-landing-system : lando
This "annotations" report provides a different view of manifest data,
concentrating on the manifest annotations, like "skip-if". What conditions
are used in manifests? How many times does each occur?
Differential Revision: https://phabricator.services.mozilla.com/D52926
--HG--
extra : moz-landing-system : lando
Very minor enhancement to 'mach test-info tests <test-name>' such that the list of
related bugs is displayed earlier, before any of the ActiveData queries, so that
all the quickly-retrieved info is displayed before the annoying ActiveData delay.
I had to split set_test_name() into two parts since the first part is required
for the bugzilla report; the second part is not required for bugzilla and involves
a lengthy ActiveData query.
Differential Revision: https://phabricator.services.mozilla.com/D52087
--HG--
extra : moz-landing-system : lando
Remove some overly clever code and make the spidermonkey-related tests more
consistent with each other.
Differential Revision: https://phabricator.services.mozilla.com/D51901
--HG--
extra : moz-landing-system : lando
Standardize the jsapi-tests logic to match the other spidermonkey check
commands in testing/mach_commands.py.
Differential Revision: https://phabricator.services.mozilla.com/D51530
--HG--
extra : moz-landing-system : lando
This allows running the jit_test.py harness using mach. It works very
similarly to the existing 'mach jstests' command.
Differential Revision: https://phabricator.services.mozilla.com/D51518
--HG--
extra : moz-landing-system : lando
Treat the reftest 'fails-if' as a synonym for manifest parser 'fail-if'.
For any annotation types requested by --filter-keys, display those annotations
in the test-level report (--show-tests).
Differential Revision: https://phabricator.services.mozilla.com/D51166
--HG--
extra : moz-landing-system : lando
Pull calls to mozbuild_reader().files_info() out of loops, instead passing
all the paths of interest at once.
Differential Revision: https://phabricator.services.mozilla.com/D49358
--HG--
extra : moz-landing-system : lando
Adds 'tests' sub-command for 'mach test-info'.
'mach help test-info' shows all sub-commands.
'mach help test-info <subcommand>' shows sub-command descriptions,
including help for the options previously excluded.
Differential Revision: https://phabricator.services.mozilla.com/D48079
--HG--
extra : moz-landing-system : lando
Instead of choosing specific run.types of interest, like 'e10s', concatenate
all returned strings, eliminating known exceptions, like 'chunked'. Notably
this correctly displays "linux64/debug-e10s-service-worker".
Differential Revision: https://phabricator.services.mozilla.com/D48091
--HG--
extra : moz-landing-system : lando
Report both failures and skipped test counts.
(Also incidentally fixes platform name reporting of fission tests.)
Differential Revision: https://phabricator.services.mozilla.com/D44074
--HG--
extra : moz-landing-system : lando
Ensure that if we try to create a mochitest of unsupported file type we return a
useful error
Differential Revision: https://phabricator.services.mozilla.com/D38224
--HG--
extra : moz-landing-system : lando
If test is run using the `mach test` harness, populate the `extra_args` with `--debugger=<name>` that is expected by the underlying harnesses (`wpt,`, `mochitest`) in order to maintain compatibility.
Differential Revision: https://phabricator.services.mozilla.com/D42952
--HG--
extra : moz-landing-system : lando
Optional summary provides a convenient synopsis of matching tests. filter-values regex
was requested for in bug 1568638, like --filter-values='(?<!!)fission'.
Differential Revision: https://phabricator.services.mozilla.com/D42570
--HG--
extra : moz-landing-system : lando
Keep the existing report format by default, but provide an option to generate a
report that is not broken down by component.
Differential Revision: https://phabricator.services.mozilla.com/D41427
--HG--
extra : moz-landing-system : lando
These were already supported by the test resolver; let's use them. More tweaks planned...
Differential Revision: https://phabricator.services.mozilla.com/D41414
--HG--
extra : moz-landing-system : lando
Basic test infrastructure for Fluent migration recipes, automate creating
a reference from recipe data, and running the recipe dry and wet.
There are no hard failures for diffs, as it might just be OK.
Differential Revision: https://phabricator.services.mozilla.com/D40200
--HG--
extra : moz-landing-system : lando
This provides the basic functionality, but certainly not all the bells and whistles. Future work may add summaries and totals, better reftest support, better web-platform support, more flexible filtering, etc.
Differential Revision: https://phabricator.services.mozilla.com/D39224
--HG--
extra : moz-landing-system : lando
To not make jstests even slower, this adds --jitflags=jstests and uses it for
jstests in automation. For jstests we can use fewer jit configurations than
for jit-tests.
This also removes the bigint and bigintdebug autospider variants. BigInt is now
enabled by default so we no longer need the configure flag etc.
Differential Revision: https://phabricator.services.mozilla.com/D36307
--HG--
extra : moz-landing-system : lando
Also adds it to the mach command, which is a little weird, because the
mach command doesn't expose the option but does parse it via the cpp unit
argument parser. So I just exposed it on the mach command and after that
it Just Works for mach.
Differential Revision: https://phabricator.services.mozilla.com/D35859
--HG--
extra : moz-landing-system : lando
Follow-up on ekyle's kind suggestion. I also found the type of some of the
returned data has changed, producing incorrect zero counts. A simple type
conversion gets everything working again:
$ ./mach test-info long-tasks
Tasks nearing their max-run-time on mozilla-central,mozilla-inbound,autoland between 2019-06-03 and 2019-06-10
test-linux64-asan/opt-web-platform-tests-e10s-10 : 52 of 212 runs (24.0%) exceeded 90% of max-run-time (7200 s)
test-linux64/debug-telemetry-tests-client-e10s : 34 of 211 runs (16.0%) exceeded 90% of max-run-time (600 s)
test-linux64-asan/opt-telemetry-tests-client-e10s : 76 of 696 runs (10.0%) exceeded 90% of max-run-time (600 s)
Differential Revision: https://phabricator.services.mozilla.com/D34362
--HG--
extra : moz-landing-system : lando
The format of platform and type data seems to have evolved over time; these
changes seem to handle the new formats well.
Differential Revision: https://phabricator.services.mozilla.com/D28766
--HG--
extra : moz-landing-system : lando
This officially makes 'moztest.resolve' the source of truth when it comes to
suite names. It aligns that file with the names used in both the
desktop_unittest and android_emulator_unittest scripts.
Differential Revision: https://phabricator.services.mozilla.com/D27555
--HG--
extra : moz-landing-system : lando
Add addtest support for per-suite arguments and multiple files. Also
support opening the created test in an editor. This allowed supporting
the wpt suite and replaces `mach wpt-create`.
# Create a wpt test
./mach addtest testing/web-platform/tests/accelerometer/test.html
# Create a wpt reftest
./mach addtest --suite wpt-reftesttesting/web-platform/tests/css/example.html --ref example-ref.html
The files created will be opened in the default editor if --editor is
supplied or a specified editor if the argument is given a value.
Differential Revision: https://phabricator.services.mozilla.com/D26339
--HG--
extra : moz-landing-system : lando
Instead of:
perl gen_template.pl -b=123456 --type=plain > path/to/test_bug123456.html
You can do:
./mach addtest --suite="mochitest-plain" > path/to/test_bug123456.html
But you can also pass in a new file path and let it guess the suite/doc:
```
# mochitest-chrome tests
./mach addtest js/xpconnect/tests/chrome/test_chrome.html
./mach addtest js/xpconnect/tests/chrome/test_chrome.xhtml
./mach addtest js/xpconnect/tests/chrome/test_chrome.xul
# mochitest-plain tests
./mach addtest js/xpconnect/tests/mochitest/test_plain.html
./mach addtest js/xpconnect/tests/mochitest/test_plain.xhtml
./mach addtest js/xpconnect/tests/mochitest/test_plain.xul
# mochitest-browser tests
./mach addtest browser/base/content/test/alerts/browser_foo.js
# xpcshell tests
./mach addtest browser/components/extensions/test/xpcshell/test_xpcshell.js
```
This also changes the mochitest template files in the following ways:
- removes the bug # boilerplate
- remove some unnecessary attributes in the template
- removes the th.template
- adds the browser.template for browser-chrome tests
Differential Revision: https://phabricator.services.mozilla.com/D25482
--HG--
rename : testing/mochitest/static/chrome.template.txt => testing/mochitest/static/chromehtml.template.txt
rename : testing/mochitest/static/test.template.txt => testing/mochitest/static/plainhtml.template.txt
rename : testing/mochitest/static/xhtml.template.txt => testing/mochitest/static/plainxhtml.template.txt
rename : testing/mochitest/static/xul.template.txt => testing/mochitest/static/plainxul.template.txt
extra : moz-landing-system : lando
Run with:
./mach jsshell-bench ares6
This requires a non-artifact build with ac_add_options --enable-js-shell. Otherwise,
a custom JS shell can be specified with --binary.
MozReview-Commit-ID: C8WAxG9pIkk
--HG--
extra : rebase_source : 99163261b771bfb2f903892f4c1260c8b241ce71
This sets verbose=True (used by the mach formatter) and compact=False (used by tbplformatter) if
only a single test is specified with either |mach test| or |mach mochitest|.
This way all TEST_STATUS and log lines will be visible when developers are writing a new test.
MozReview-Commit-ID: 2nuKs9DLnx0
--HG--
extra : rebase_source : 1fc594b465a2a94dfcd85d56d042969af47f6f65
This was originally supposed to be fixed in bug 1421799, but I either forgot to
switch the flag, or accidentally clobbered it while resolving conflicts.
MozReview-Commit-ID: Im8OKbswEf0
--HG--
extra : rebase_source : 1104969aca56b62f8f302225fd11a509a73a3960
The end goal here is to be able to use |mach try fuzzy <path>| with tests that
belong to a subsuite. To do this, we need a unique 'task_regex' value for each
subsuite so that we can map a test path back to a set of tasks.
This removes the TEST_FLAVORS dict (which was mostly just a redefinition of the
data in TEST_SUITES), and instead provides two new private mappings:
<flavor> -> suite definition
(<flavor>, <subsuite>) -> suite definition
To retrieve a suite definition given a flavor/subsuite, consumers can now call
get_suite_definition.
MozReview-Commit-ID: 2pe1v1IHUVy
--HG--
extra : rebase_source : 6fff947ba214112ccf16c894174a6a0e2487111a
When 'summary_on_shutdown' is True (which is the case for |mach test| and |mach
mochitest|), BaseSummaryFormatters will save the summary information until the
'shutdown' action is received at the end of the logger's lifetime.
Summary information will no longer be dumped on 'suite_end'.
MozReview-Commit-ID: HKtVr5PxfOy
--HG--
extra : rebase_source : f350f09111deb510b27a4e55797243dda3160869
In order for |mach test| and |mach mochitest| to log an overall summary,
every test harness invocation they make needs to use the same structured
logger (otherwise the affected suite will be missing from the summary).
MozReview-Commit-ID: 8LJw7r8SItk
--HG--
extra : rebase_source : 1417dce3817bae94ad61a5250065c6cbc35857e4
The code in |mach test| for test resolving, should get merged with the TestResolver
class in moztest.resolve. This way it can be shared with other modules and we'll
have a single canonical place for all our test resolving logic.
MozReview-Commit-ID: IHRXXi5mB4G
--HG--
extra : rebase_source : 6f96d06412ab8fa152ac5d9bdd15acbcdc9695c4
The TestMetadata and TestResolver classes aren't technically part of the build
system. The only connection is that they consume some build system output.
The next patch in this series is going to be merging in a bunch of other test
resolving logic from other parts of the tree. Moving this out first allows us
to keep that extra logic out of mozbuild.
MozReview-Commit-ID: 1eq4SjFVCyW
--HG--
rename : python/mozbuild/mozbuild/test/test_testing.py => testing/mozbase/moztest/tests/test_resolve.py
extra : rebase_source : 7ff11f9ec455547533082d20cb5371845f7a4f21
find_paths_and_tags() can instantiate a BuildReader to read moz.build
metadata. We switch `mach test` to call this to avoid some redundant
code.
To facilitate `mach try`'s use case, we add test flavors to the
result of find_paths_and_metadata() and change the function name to
reflect its more generic purpose.
MozReview-Commit-ID: Lcz05W8g2dt
--HG--
extra : rebase_source : 17d6f23275f4eed1bd544a9b2cd991214625dabc
This will make the return value more easily extensible and will help
consumers know what the data structures represent.
MozReview-Commit-ID: DaeYsqfMW37
--HG--
extra : rebase_source : 440d16f0604b21f0ec885e1bca5066c32b18f641
The AutoTry code got moved around in 27b5181a3c77 and c94bf66df122
(bug 1382775). This broke `mach test` without arguments.
This commit imports the new location and uses the new API. As the
inline TODO states, there is still room for improvement. This will
be done in a separate commit.
MozReview-Commit-ID: AKHfdNnfOhl
--HG--
extra : rebase_source : dedd0171bafc1897c71f0710addb1f692c9f4844
Now that we have a helper function to obtain a BuildReader, let's
put it to use.
MozReview-Commit-ID: 7V3RsWs5TPu
--HG--
extra : rebase_source : 23193a1482ebb2fc4d1bdc588d8cd31c4d458645
This is a pretty bad runner. My goal for now is to get something really
basic working and then improve it over time as more tests get added.
Run tests with:
./mach cramtest path/to/dir
Any extra args will be forwarded to the cram binary, e.g:
./mach cramtest -i path/to/dir
Cram help can be seen with:
./mach python -m cram -- -h
MozReview-Commit-ID: 67jYHfYQjWu
--HG--
extra : rebase_source : 1ee1dc5ddcdc4559402be019996bd757afe824ab
This was mostly generated with autopep8 v1.3.2. A few left over errors
were fixed by hand.
MozReview-Commit-ID: 5crfUz0xj3O
--HG--
extra : rebase_source : 1fa9f550dbdca35ce4f2d8f9848dc3061b7fe01b
Recent changes in bug 1377216 introduced some new Python imports
to some SpiderMonkey test scripts. These modules likely won't be on
sys.path by default.
To ensure these Python processes always work, this commit changes
them to be executed with the Python binary from the build system's
virtualenv.
In the case of jit_test.py and jstests.py, we switch from relying on
the shebang Python (`/usr/bin/env python`). In other cases, we switch
from mach's python (essentially `which python2.7` or `python`).
Previously, some clients may have been using Python 3 via shebang
discovery. This change will force them to Python 2. This might be
unfortunate. But proper Python 3 support in mach and the build system
is a bit of a mess right now. I'd rather we just consistently use
the virtualenv Python because it is more well-defined and it
eliminates a class of bugs around the Python environment being
undefined.
MozReview-Commit-ID: 4NHzWisIgDC
--HG--
extra : rebase_source : cb79fe457daf7780aabcef902718429fdd5072a5
The |mach try| implementation isn't really testing specific so figured while doing this refactor
anyway we could take the opportunity to move it to /tools. This renames the "autotry" module to
"syntax" as one of the 'selectors' under the newly created tryselect module.
MozReview-Commit-ID: DmnGU5sKuCk
--HG--
rename : testing/tools/autotry/autotry.py => tools/tryselect/selectors/syntax.py
extra : rebase_source : 52f2a61d97107b8d50e3894ea0f5cd9c24d4108d
The -j try syntax specifies a "job" task. Job tasks are different from
unittests in that they typically don't require a build and run from the
source tree. Examples include linters (eslint, flake8, etc), python tests
(mozbase, mochitest, etc) and misc things like doc generation.
Sometimes, developers might only want to run a specific "job" task with a
syntax like "./mach try -j mozbase".
This means a few assumptions need to be broken in |mach try|. Platforms
and unittests should no longer be required if -j was specified. Most of
the changes in this patch revolve around dealing with those broken
assumptions.
MozReview-Commit-ID: O0y6V2Wwej
--HG--
extra : rebase_source : d003380317cbb688fcc7113c454a5c0543ade40c
Currently if no unittests, paths or tags are specified it will error out.
MozReview-Commit-ID: O0y6V2Wwej
--HG--
extra : rebase_source : e7a438aa1131e3d473459c46fbe9f23c3c06bae1
sutagent is no longer built or used; devicemanagerSUT is completely
unused. After this change, devicemanagerADB is the only implementation of
devicemanager, and test harness options like --dm_trans are eliminated.
Ricky Stewart
Andrew Halberstadt
Geoff Brown
Tooru Fujisawa
Jan de Mooij
Hamzah
Chris Manchester
Gijs Kruitbosch
Botond Ballo
Ted Campbell
Cosmin Sabou
James Graham
Mihai Alexandru Michis
Edwin Gao
Axel Hecht
Kartikaya Gupta
Aaron Klotz
Brian Grinstead
Julian Descottes
Ms2ger
Sebastian Hengst
Dorel Luca
Attila Craciun
Gregory Szorc
Kris Maglione