Makes it possible for mach to resolve test paths for Puppeteer,
so that individual tests can be run from the command line using
"./mach test", as such:
% ./mach test remote/test/puppeteer/test/screenshot.spec.js
As the Puppeteer test suite is imported from upstream and we cannot
change this directory at will (i.e. to add test manifest files),
we take the same approach as for WPT and populate the manifest by
recursively walking the remote/test/puppeteer/test/**/*.spec.js file tree.
Differential Revision: https://phabricator.services.mozilla.com/D37013
--HG--
extra : moz-landing-system : lando
Makes it possible for mach to resolve test paths for Puppeteer,
so that individual tests can be run from the command line using
"./mach test", as such:
% ./mach test remote/test/puppeteer/test/screenshot.spec.js
As the Puppeteer test suite is imported from upstream and we cannot
change this directory at will (i.e. to add test manifest files),
we take the same approach as for WPT and populate the manifest by
recursively walking the remote/test/puppeteer/test/**/*.spec.js file tree.
Differential Revision: https://phabricator.services.mozilla.com/D37013
--HG--
extra : moz-landing-system : lando
Previously we would silently change the value of "e10s" from False to True.
This can cause confusion and lead people to falsely think mochitest-chrome/a11y
work with e10s (they do not).
Now we explicitly error out in this case. This might be slightly less
convenient for the developer (e.g they might need to re-run the command), but
the downside of needing to rerun a test command is less than the risk of
misunderstanding what is being tested.
Note: when running |mach test| or |mach mochitest| on a directory that contains
both chrome/a11y and another suite, we'll still do the right thing and
implicitly set "e10s=False".
Differential Revision: https://phabricator.services.mozilla.com/D28538
--HG--
extra : moz-landing-system : lando
This officially makes 'moztest.resolve' the source of truth when it comes to
suite names. It aligns that file with the names used in both the
desktop_unittest and android_emulator_unittest scripts.
Differential Revision: https://phabricator.services.mozilla.com/D27555
--HG--
extra : moz-landing-system : lando
Since e10s is the default configuration, we shouldn't explicitly mark things
with the "-e10s" suffix. Instead we should mark things that *don't* run with
'e10s. This patch removes '-e10s' from all treeherder group symbols and task
labels, adds the "-1proc" suffix to tasks that are non-e10s.
Differential Revision: https://phabricator.services.mozilla.com/D25958
--HG--
extra : moz-landing-system : lando
If mozbuild parsing fails due to a missing file (eg: a file not existing
in UNIFIED_SOURCES), then no Makefiles are written out, but
config.status exists. This would cause mozbuild to think that configure
doesn't need to run, and rely on make to perform the backend-out-of-date
check in rebuild-backend.mk. Unfortunately since no Makefiles were
written, the make command fails immediately and no attempt is made to
re-create the backend. Note that this is only a problem if the first
mozbuild parsing from a clobber build fails, otherwise there is
typically a top-level Makefile from a previous build to call into (at
which point make can determine it is out-of-date, and re-invoke itself).
The fix is to have the RecursiveMake backend re-use the same logic that
was introduced into mozbuild for alternate backends, and remove
rebuild-backend.mk. This way, mozbuild can always determine if the
backend needs to be regenerated, even if the initial parsing failed.
Test code was also relying on rebuild-backend.mk to generate the
TestBackend, but moving backend_out_of_date() into MozbuildObject allows
this code to be shared.
Differential Revision: https://phabricator.services.mozilla.com/D26262
--HG--
rename : build/gen_test_backend.py => python/mozbuild/mozbuild/gen_test_backend.py
extra : moz-landing-system : lando
If mozbuild parsing fails due to a missing file (eg: a file not existing
in UNIFIED_SOURCES), then no Makefiles are written out, but
config.status exists. This would cause mozbuild to think that configure
doesn't need to run, and rely on make to perform the backend-out-of-date
check in rebuild-backend.mk. Unfortunately since no Makefiles were
written, the make command fails immediately and no attempt is made to
re-create the backend. Note that this is only a problem if the first
mozbuild parsing from a clobber build fails, otherwise there is
typically a top-level Makefile from a previous build to call into (at
which point make can determine it is out-of-date, and re-invoke itself).
The fix is to have the RecursiveMake backend re-use the same logic that
was introduced into mozbuild for alternate backends, and remove
rebuild-backend.mk. This way, mozbuild can always determine if the
backend needs to be regenerated, even if the initial parsing failed.
Test code was also relying on rebuild-backend.mk to generate the
TestBackend, but moving backend_out_of_date() into MozbuildObject allows
this code to be shared.
Differential Revision: https://phabricator.services.mozilla.com/D26262
--HG--
rename : build/gen_test_backend.py => python/mozbuild/mozbuild/gen_test_backend.py
extra : moz-landing-system : lando
Some of these were working with the '<flavor>-<subsuite>' mechanism that was
previously being used, but better to be explicit wherever possible.
Depends on D25077
Differential Revision: https://phabricator.services.mozilla.com/D25078
--HG--
extra : moz-landing-system : lando
We should use filter functions instead of regexes here. Would be a lot more robust.
Depends on D25076
Differential Revision: https://phabricator.services.mozilla.com/D25077
--HG--
extra : moz-landing-system : lando
Test manifests may be included by multiple other manifests, optionally
with additional variables below the `[include:...]` section header.
These additional variables are specific to the manifest that contained
the "include" section, and should not inadvertently be shared with other
manifests that also happen to include this manifest.
To achieve that, store the defaults for included manifests in a (path to
parent manifest, path to included manifest) tuple instead of just the
included manifest.
Differential Revision: https://phabricator.services.mozilla.com/D18086
--HG--
extra : moz-landing-system : lando
Test manifests may be included by multiple other manifests, optionally
with additional variables below the `[include:...]` section header.
These additional variables are specific to the manifest that contained
the "include" section, and should not inadvertently be shared with other
manifests that also happen to include this manifest.
To achieve that, store the defaults for included manifests in a (path to
parent manifest, path to included manifest) tuple instead of just the
included manifest.
Differential Revision: https://phabricator.services.mozilla.com/D18086
--HG--
extra : moz-landing-system : lando
The following python-test paths are being moved out of 'make check' and into their own task:
- python/mozlint
- testing/mozbase
- tools/lint
The following python-test paths previously did not run on Windows:
- python/mozterm
- testing/marionette
- testing/raptor
- tools/tryselect
Differential Revision: https://phabricator.services.mozilla.com/D10759
--HG--
extra : moz-landing-system : lando
The following python-test paths are being moved out of 'make check' and into their own task:
- python/mozlint
- testing/mozbase
- tools/lint
The following python-test paths previously did not run on Windows:
- python/mozterm
- testing/marionette
- testing/raptor
- tools/tryselect
MozReview-Commit-ID: C07FANaYzf7
Depends on D10758
Differential Revision: https://phabricator.services.mozilla.com/D10759
--HG--
extra : moz-landing-system : lando
The previous code split a lot of logic between the update and download
parts, and exposed two different mach commands. In order to simplify
things it's better to have a single entry point for manifest download
and update, and ensure that's called consistently.
Differential Revision: https://phabricator.services.mozilla.com/D7497
These regexes are used for things like determining which tasks to run given a
"path" int |mach try|. Previously, we used patterns like:
mochitest-chrome-(?:e10s)?(?:-1)?$
This would match both e10s and non-e10s versions of a task with either no
chunks, or only selecting chunk 1. But we keep adding other configurations, e.g
-gpu, -no-accel, -sw, etc. Each time we create a new possibility we need to
remember to update these task regexes (or else lose test coverage when using
paths with |mach try|).
Instead of individually listing every possibility, let's use a pattern like
this:
mochitest-chrome($|.*(-1|[^0-9])$)
This also selects tasks that are either chunk 1 or don't have any chunks. But
it allows for arbitrary strings in-between. This regex doesn't need to be
updated when we add configurations like -sw.
Depends on D7119
Differential Revision: https://phabricator.services.mozilla.com/D7120
--HG--
extra : moz-landing-system : lando
I almost forgot to update the regexes in moztest.resolve when creating the -sw
variant of task. This adds a test to make sure we don't forget more things in
the future.
Differential Revision: https://phabricator.services.mozilla.com/D7119
--HG--
extra : moz-landing-system : lando
mach try fuzzy has some stronger requirements compared to mach test.
Depends on D6673
Differential Revision: https://phabricator.services.mozilla.com/D6674
--HG--
extra : moz-landing-system : lando
Changes the wpt manifest path to the topobjdir instead so it can be moved out of tree.
Other changes so that the manifest download and update, and |mach wpt| and |mach test <wpt-test>| work with the new path.
The manifest is also downloaded and updated when creating the tests-archive to ensure that it exists when we run tests on TC.
MozReview-Commit-ID: Fp6UsKJjhTU
Differential Revision: https://phabricator.services.mozilla.com/D5312
--HG--
extra : moz-landing-system : lando
Changes the wpt manifest path to the topobjdir instead so it can be moved out of tree.
Other changes so that the manifest download and update, and |mach wpt| and |mach test <wpt-test>| work with the new path.
The manifest is also downloaded and updated when creating the tests-archive to ensure that it exists when we run tests on TC.
MozReview-Commit-ID: Fp6UsKJjhTU
Differential Revision: https://phabricator.services.mozilla.com/D5312
--HG--
extra : moz-landing-system : lando
It's unclear how this situation can actually happen, but it has caused a couple of
intermittents.
Differential Revision: https://phabricator.services.mozilla.com/D2355
--HG--
extra : moz-landing-system : lando
Also include webgl2-deqp, which we would like to run eventually, but not yet.
MozReview-Commit-ID: CY4hYCI95ws
--HG--
extra : rebase_source : 9973df0f905bb65d2e8b8c66a6a57e8869e527c1
Andreas Tolfsen
Ciure Andrei
Andrew Halberstadt
Alexandre Poirot
b4hand
Narcis Beleuzu
Sebastian Hengst
Joel Maher
Mike Shal
Andreea Pavel
Rob Wu
Csoregi Natalia
Wes Kocher
Marco Castelluccio
James Graham
Ahilya Sinha
shindli
Chris AtLee
Bogdan Tara
Jeff Gilbert