Changes:
- add `jsreftest` to the list of permitted tests to run on `android-hw-p2-8-0-arm7-api-16`
- add necessary mozharness setup for `android-hw` for jsreftest
- disable `android-em` from running jsreftests
Differential Revision: https://phabricator.services.mozilla.com/D32262
--HG--
extra : moz-landing-system : lando
Changes:
__mochitest__
- demote `android-em` to tier 2
- promote `android-hw` to tier 1
- expand regex for `run-on-projects` specification to include all `android-em` instances, and restrict to `try`
Differential Revision: https://phabricator.services.mozilla.com/D32541
--HG--
extra : moz-landing-system : lando
Changes:
- enabled SM(p) runs on `windows10-aarch64` for `built-projects`
- turn off `jittest` runs for all platforms matching `android-hw-.*-aarch64/.*`
Differential Revision: https://phabricator.services.mozilla.com/D32220
--HG--
extra : moz-landing-system : lando
Changes:
- demote all existing `android-em-4.*` tests to tier 2
- ensure the above tests only run on `try` and `mozilla-central` but with exceptions
Differential Revision: https://phabricator.services.mozilla.com/D32086
--HG--
extra : moz-landing-system : lando
Changes:
- demote all existing `android-em-4.*` tests to tier 2
- ensure the above tests only run on `try` and `mozilla-central` but with exceptions
Differential Revision: https://phabricator.services.mozilla.com/D32086
--HG--
extra : moz-landing-system : lando
I had been waiting for more complete x86_64 coverage, but realized that's
not strictly necessary: TV works okay as-is on Android x86_64.
Differential Revision: https://phabricator.services.mozilla.com/D32347
--HG--
extra : moz-landing-system : lando
Bug 1553584 was filed and shows a race condition in the test harness that could make fixing bugs difficult. Bumping these back to tier-3 until that can be fixed.
Differential Revision: https://phabricator.services.mozilla.com/D32240
--HG--
extra : moz-landing-system : lando
Changes:
- edited `linux32-tests` to include only web-platform-tests
- removed references to other `linux32` platforms from `test-platforms.yml` other than `linux32-shippable/opt`
Differential Revision: https://phabricator.services.mozilla.com/D29792
--HG--
extra : moz-landing-system : lando
This patch runs wpt against opt and debug builds of geckoview as tier-2 jobs on mozilla-central and try.
Differential Revision: https://phabricator.services.mozilla.com/D31581
--HG--
extra : moz-landing-system : lando
These chunks are quite out of balance: Probably there are some very long running
tests in xpcshell-7 with 8 chunks. Even so, I think the easiest way to avoid timeouts
is to return to 12 chunks.
Differential Revision: https://phabricator.services.mozilla.com/D31685
--HG--
extra : moz-landing-system : lando
We can run /debug mochitests against geckoview for the cost of another dozen
or so test annotations. Both /opt and /debug mochitests are nearly worthy of
tier 1, but still waiting for bug 1534732.
Differential Revision: https://phabricator.services.mozilla.com/D30931
--HG--
extra : moz-landing-system : lando
Changes:
- set `macosx64-ccov` tests to 8 chunks (unchanged from m-c)
- set windows and linux `ccov` to 6 chunks (from 8)
- set `linux64.*/debug` to 6 chunks (from 10 and 8)
- set `android-em` to 8 chunks (unchanged)
- set default chunks to be 5 (from 8)
Differential Revision: https://phabricator.services.mozilla.com/D29713
--HG--
extra : moz-landing-system : lando
Changes:
- change the default chunk count to 4 (from 6)
- increase `ccov` chunk count to 8 (from 6)
- `qr` and `linux64/debug` are to remain at 6 chunks
- reduce all other `linux` chunks to 3 (from 6)
- `windows/debug` (including `qr`) chunks reduced to 5 (from 6)
Default chunk count of 4 for Windows and macOS are conservative, leading to sub-20 minute runtimes for most chunks.
Differential Revision: https://phabricator.services.mozilla.com/D29875
--HG--
extra : moz-landing-system : lando
Changes:
- reduced `linux64/debug` chunking to 14 (from 16)
- reduced `linux64-ccov/debug` to 14 (from 16)
- increased `windows64-ccov` to 16 (from 10)
- maintain all non-ccov debug to 8
- maintain all asan to 8
- reduce everything else to 5
Differential Revision: https://phabricator.services.mozilla.com/D30303
--HG--
extra : moz-landing-system : lando
Changes:
- reduced `linux64/debug` chunking to 14 (from 16)
- reduced `linux64-ccov/debug` to 14 (from 16)
- increased `windows64-ccov` to 16 (from 10)
- maintain all non-ccov debug to 8
- maintain all asan to 8
- reduce everything else to 5
Differential Revision: https://phabricator.services.mozilla.com/D30303
--HG--
extra : moz-landing-system : lando
Similar to bug 632954, this disables the Android aarch64 tests on opt
except on try, and instead runs the tests on Android aarch64 pgo builds.
Differential Revision: https://phabricator.services.mozilla.com/D29589
--HG--
extra : moz-landing-system : lando
Changes:
- disable wholesale the `media-source` subsuite within web-platform-test
Attempts were made to isolate the tests that cause the test harness to lock up with a permission issue on `Ahem.ttf` however the attempts were unsuccessful in isolating the tests to a manageable set. It appears the solution is to simply stop running `media-source` tests on windows10-aarch64 as that seems to have solved the problem in a previous try run.
Differential Revision: https://phabricator.services.mozilla.com/D28778
--HG--
extra : moz-landing-system : lando
- stop mochitest-headless on windows10
- stop mochitest-headless on linux64/debug
- make mochitest-headless tier-2
- make mochitest-headless run on m-c/try
Differential Revision: https://phabricator.services.mozilla.com/D28715
--HG--
extra : moz-landing-system : lando
This patch leaves wpt running against fennec on androidx86 as tier2, adds wpt to run against geckoview testactivity on android x86_64 as tier3, and adds enough metadata to run_info_extras to help differentiate the two in expectation files. Fennec is "os == android and not e10s", while geckoview is "os == android and e10s".
Differential Revision: https://phabricator.services.mozilla.com/D27182
--HG--
extra : moz-landing-system : lando
This patch leaves wpt running against fennec on androidx86 as tier2, adds wpt to run against geckoview testactivity on android x86_64 as tier3, and adds enough metadata to run_info_extras to help differentiate the two in expectation files. Fennec is "os == android and not e10s", while geckoview is "os == android and e10s".
Differential Revision: https://phabricator.services.mozilla.com/D27182
--HG--
extra : moz-landing-system : lando
This officially makes 'moztest.resolve' the source of truth when it comes to
suite names. It aligns that file with the names used in both the
desktop_unittest and android_emulator_unittest scripts.
Differential Revision: https://phabricator.services.mozilla.com/D27555
--HG--
extra : moz-landing-system : lando
Currently we have the concept of a "suite" and a "flavour" in our task
configuration. Typically, the "suite" refers to the high-level test harness
like "mochitest" or "reftest", whereas the flavour is more specific, e.g
"browser-chrome-instrumentation" or "crashtest". However the line between suite
and flavour is not applied with any semblance of consistency which results in
inconsistent naming throughout the tree.
This patch gets rid of the concept of "flavours" entirely (at least when it
comes to task configuration). A suite is a type of test run, for example:
- mochitest-plain
- mochitest-devtools-chrome
- mochitest-browser-chrome-instrumentation
- jsreftest
- reftest
- firefox-ui-functional-remote
etc
There is no confusion here between suites and flavours because flavours don't
exist. However, there are a couple of places where we *do* need to know what
"test harness" is used to run a suite. These cases are:
1. For SCHEDULES moz.build rules
2. For the desktop_unittest.py mozharness script which takes arguments like
--mochitest-suite=browser (this is not a compelling use of this information
and should be refactored to work more like the android_emulator_unittest.py
script)
So to get this information, this patch introduces a new concept of a "category"
which is the overall "test harness" that runs the suite. For many suites, the
"category" is identical to the suite name. Unlike flavours, "categories" have
no bearing on how we call or refer to the suite.
Differential Revision: https://phabricator.services.mozilla.com/D27554
--HG--
extra : moz-landing-system : lando
Turns out these suites were hardcoded to be non-e10s in the mochitest harness.
So while it looked like they were working with e10s in treeherder, they were
actually still running with it disabled.
Turning e10s on causes both suites to permafail due to timeouts.
Depends on D28386
Differential Revision: https://phabricator.services.mozilla.com/D28387
--HG--
extra : moz-landing-system : lando
This patch leaves wpt running against fennec on androidx86 as tier2, adds wpt to run against geckoview testactivity on android x86_64 as tier3, and adds enough metadata to run_info_extras to help differentiate the two in expectation files. Fennec is "os == android and not e10s", while geckoview is "os == android and e10s".
Differential Revision: https://phabricator.services.mozilla.com/D27182
--HG--
extra : moz-landing-system : lando
We are starting to spin off more and more "variants" of test suites. These are
usually just duplicates of our pre-existing tasks, except with an additional
pref set.
Currently there are two variants (serviceworker-e10s and socketprocess-e10s),
but a third will be added soon (fission). This change ensures we handle these
types of requests in a consistent and well defined manner. It also splits tasks
in a loop, so we don't accidentally risk combinatorial explosion.
Variants should typically be reserved for very large changes that will impact
the entire codebase (think e10s).
Differential Revision: https://phabricator.services.mozilla.com/D28061
--HG--
extra : moz-landing-system : lando
Changes:
- most tests are skipped using `moz.build` configuration file.
- `MultiWriterQueue` had to be skipped with `define` clauses in the test file due to build bustages when its `moz.build` file was used.
Differential Revision: https://phabricator.services.mozilla.com/D27944
--HG--
extra : moz-landing-system : lando
Many tasks (release tasks and cached tasks, in particular) should be re-run rather
than retriggered. Disable retrigger action for those tasks by default.
Differential Revision: https://phabricator.services.mozilla.com/D27206
--HG--
extra : moz-landing-system : lando
Reduce chunks from 8 to 3. Each test task has at least a couple of minutes
of overhead, so fewer chunks improves overall efficiency. At 3 chunks, each
one still completes reasonably quickly (less than 20 minutes).
Differential Revision: https://phabricator.services.mozilla.com/D27339
--HG--
extra : moz-landing-system : lando
Stop running Tss(tp6) and T(bcv) on ccov builds on central -- the remaining 2 cases
missed in the previous bug.
Differential Revision: https://phabricator.services.mozilla.com/D27313
--HG--
extra : moz-landing-system : lando
Add Android 7.0 gtests, opt and debug, running against the geckoview
TestRunnerActivity.
Differential Revision: https://phabricator.services.mozilla.com/D27016
--HG--
extra : moz-landing-system : lando
So, instead of fetches['by-test-platform']['fetch'], we have
fetches['fetch']['by-test-platform'].
Differential Revision: https://phabricator.services.mozilla.com/D27227
--HG--
extra : moz-landing-system : lando
Enables the list of suites as defined in Bug 1540213.
- added new item in `test-sets.yml` for windows10-aarch64.
- reference the new test-set in `test-platforms.yml`.
Differential Revision: https://phabricator.services.mozilla.com/D25979
--HG--
extra : moz-landing-system : lando
These will run on android-hw against the latest reference browser nightly.
Since they are try-only, they can only be scheduled with |mach try fuzzy
--full|.
Differential Revision: https://phabricator.services.mozilla.com/D25801
--HG--
extra : moz-landing-system : lando
This patch introduces a new marionette media test along
with a Youtube test.
To run the Youtube streaming test locally:
./mach marionette-test dom/media/test/marionette/test_youtube.py -vv --gecko-log -
Differential Revision: https://phabricator.services.mozilla.com/D23644
--HG--
extra : moz-landing-system : lando
This patch introduces a new marionette media test along
with a Youtube test.
To run the Youtube streaming test locally:
./mach marionette-test dom/media/test/marionette/test_youtube.py -vv --gecko-log -
Differential Revision: https://phabricator.services.mozilla.com/D23644
--HG--
extra : moz-landing-system : lando
An `sy-tp6` variant is added to the AWSY test suite that runs against the tp6 pageset.
Differential Revision: https://phabricator.services.mozilla.com/D24585
--HG--
extra : moz-landing-system : lando
This also renames win_taskcluster_unittest.py to win_unittest.py for
consistency with the other platforms.
Differential Revision: https://phabricator.services.mozilla.com/D25401
--HG--
extra : moz-landing-system : lando
Jmaher indicated we do not have the test capacity to incur this as a duplicated set
Differential Revision: https://phabricator.services.mozilla.com/D23382
--HG--
extra : moz-landing-system : lando
This also relates to Bug 1522111 where we turned off opt tests in favor of pgo,
shippable is like the new pgo so do that.
This has a side affect of adding talos-tp6-stylo-threads to inbound for
osx-shippable (previously was only on autoland).
This has no practical affect after D23382 is applied (because of lack of OSX capacity)
Differential Revision: https://phabricator.services.mozilla.com/D23129
--HG--
extra : moz-landing-system : lando
Should there end up being a need we can back out this patch and let them run, but :jmaher indicated
he was happy with dropping them entirely and not duplicating.
Differential Revision: https://phabricator.services.mozilla.com/D23383
--HG--
extra : moz-landing-system : lando
I did a bunch of manual testing with this, the biggest uncertainties lie around beta and central/nightly.
We are adding shippable-qr to beta because of replacing nightly too.
Autoland and inbound should have the same sets of tasks.
beta -
- adds mochitest-plain-headless-{1..4} to beta (not currently run)
- adds raptor to run on shippable for beta
- currently runs on opt on beta, and for nightly tasks on beta only webaudio-chrome runs.
- adds talos to shippable tasks, on beta talos only runs against opt.
central -
- adds browser-screenshots to nightly graph
- adds mochitest-plain-headless-{1..4} to nightly graph
- adds browser-instrumentation to shippable
Differential Revision: https://phabricator.services.mozilla.com/D23122
--HG--
extra : moz-landing-system : lando
Effectively back out much of the run on projects from D22710
This also has the added affect of scheduling the shippable builds to run because of dependencies.
Differential Revision: https://phabricator.services.mozilla.com/D22833
--HG--
extra : moz-landing-system : lando
raptor-chrome is Google Chrome and only needs to run once per day, so mozilla-central pushes and try.
raptor-profiling is primarily for devs to have up to date profile information and it too only needs to run once per day.
TODO is to try and find a clean way to make them only run when we trigger Nightlies rather than every m-c push.
Differential Revision: https://phabricator.services.mozilla.com/D22832
--HG--
extra : moz-landing-system : lando
This runs crashtests and reftests on ASAN builds with WR enabled, so
that we catch any ASAN regressions prior to landing without incurring
too great of a test load hit.
Differential Revision: https://phabricator.services.mozilla.com/D24952
--HG--
extra : moz-landing-system : lando
Jmaher indicated we do not have the test capacity to incur this as a duplicated set
Differential Revision: https://phabricator.services.mozilla.com/D23382
--HG--
extra : moz-landing-system : lando
This also relates to Bug 1522111 where we turned off opt tests in favor of pgo,
shippable is like the new pgo so do that.
This has a side affect of adding talos-tp6-stylo-threads to inbound for
osx-shippable (previously was only on autoland).
This has no practical affect after D23382 is applied (because of lack of OSX capacity)
Differential Revision: https://phabricator.services.mozilla.com/D23129
--HG--
extra : moz-landing-system : lando
Should there end up being a need we can back out this patch and let them run, but :jmaher indicated
he was happy with dropping them entirely and not duplicating.
Differential Revision: https://phabricator.services.mozilla.com/D23383
--HG--
extra : moz-landing-system : lando
I did a bunch of manual testing with this, the biggest uncertainties lie around beta and central/nightly.
We are adding shippable-qr to beta because of replacing nightly too.
Autoland and inbound should have the same sets of tasks.
beta -
- adds mochitest-plain-headless-{1..4} to beta (not currently run)
- adds raptor to run on shippable for beta
- currently runs on opt on beta, and for nightly tasks on beta only webaudio-chrome runs.
- adds talos to shippable tasks, on beta talos only runs against opt.
central -
- adds browser-screenshots to nightly graph
- adds mochitest-plain-headless-{1..4} to nightly graph
- adds browser-instrumentation to shippable
Differential Revision: https://phabricator.services.mozilla.com/D23122
--HG--
extra : moz-landing-system : lando
Effectively back out much of the run on projects from D22710
This also has the added affect of scheduling the shippable builds to run because of dependencies.
Differential Revision: https://phabricator.services.mozilla.com/D22833
--HG--
extra : moz-landing-system : lando
raptor-chrome is Google Chrome and only needs to run once per day, so mozilla-central pushes and try.
raptor-profiling is primarily for devs to have up to date profile information and it too only needs to run once per day.
TODO is to try and find a clean way to make them only run when we trigger Nightlies rather than every m-c push.
Differential Revision: https://phabricator.services.mozilla.com/D22832
--HG--
extra : moz-landing-system : lando
This sets all of the shippable tests to not run in the places where they would otherwise.
This patch will be effectively undone later in the patchset.
Differential Revision: https://phabricator.services.mozilla.com/D22710
--HG--
extra : moz-landing-system : lando
Linux64/debug mochitest-dt chunks are increased to 16 to provide more capacity in
reasonable run time. For ccov, mochitest-dt-12 sometimes still runs > 90 minutes,
so ccov max-run-time is increased to 2 hours.
Differential Revision: https://phabricator.services.mozilla.com/D23309
--HG--
extra : moz-landing-system : lando
This is due to an incompatability somewhere between JaCoCo and
default interface methods.
Depends on D23016
Differential Revision: https://phabricator.services.mozilla.com/D23017
--HG--
extra : moz-landing-system : lando
This was causing a JavaScript error in the |mach try chooser| interface since
the space resulted in invalid CSS selectors being used.
Differential Revision: https://phabricator.services.mozilla.com/D20710
--HG--
extra : moz-landing-system : lando
- makes changes to various configuration files, thanks jmaher
- for the time being, disable tests via `taskcluster/ci/test/test-platforms.yml` to prevent overwhelming the hardware at Bitbar
Differential Revision: https://phabricator.services.mozilla.com/D19581
--HG--
extra : moz-landing-system : lando
Enable mochitest-chrome, but skip test_bug331215.xul and test_bug304188.xul because of frequent intermittent failures on WebRener. The intermittent failures already exist and the failure seems not directly related to WebRender implementation.
Differential Revision: https://phabricator.services.mozilla.com/D19115
--HG--
extra : moz-landing-system : lando
Mitchell Hentges
alexandru.ionescu
Edwin Gao
Andrew Halberstadt
Joel Maher
Kartikaya Gupta
arthur.iakab
Geoff Brown
Wes Kocher
Henrik Skupin
Stephen Donner
Mihai Alexandru Michis
Ryan VanderMeulen
Rob Wood
Ionut Goldan
Florin Strugariu
Tom Prince
Andreea Pavel
Chris Hartjes
Bob Clary
Raphael Pierzina
Mike Shal
Brindusan Cristian
Csoregi Natalia
Mike Hommey
Cosmin Sabou
Justin Wood
Chris AtLee
Tarek Ziadé
Narcis Beleuzu
Eric Rahm
Marian Raiciof
Sebastian Hengst
Noemi Erli
Gian-Carlo Pascutto
marian.raiciof
James Willcox
Gregory Mierzwinski
Florin Strugariu
championshuttler
Byron Campen [:bwc]
Gurzau Raul
sotaro