https://github.com/xamarin/xamarin-macios/pull/4884 introduced the logic of only building certain `RunTestTask`.
This was meant to disable iOS Extensions as part of a fix to https://github.com/xamarin/maccore/issues/1008.
However this didn't quite work and iOS extensions were still running (and failing).
The reason being that `BuildOnly` was set to a `RunDeviceTask` that's added to a list which is then given to `CreateTestVariations` which creates new instances of `RunDeviceTask`.
We now propagate `BuildOnly` to the new variation instance.
This commit improves the state of the BCL testing in the following ways:
1. Improve the device tets running. Less apps, faster results.
2. WatchOS apps are left as they were to ensure that we do not have deplouyment/run issues.
We now support the ignore files per assembly name to simplify the
tracking of the ignored tests. All
* [XHarness] Ensure we do a nuget restore on bcl imported tests. Fixes#5383
We need to ensure that a nuget restore is done, otherwise we might have
build issues with missing libs.
Fixes https://github.com/xamarin/xamarin-macios/issues/5383
Most of the tests are comming from the downloaded SDK, but some of them
have to reman because they are running/passing more tests. This
differences are due to how the tests that come from mono are written
(missing resources etc..)
Please take a look at
https://devdiv.visualstudio.com/DevDiv/_workitems/edit/794210/ for the
list of bcl tests and test number differences.
It's harder to forget setting the define if it's already set by default.
Fixes the mini tests on watchOS device:
[FAIL] JitTests.Exceptions : System.Reflection.TargetInvocationException : Exception has been thrown by the target of an invocation.
----> System.ExecutionEngineException : Attempting to JIT compile method 'ExceptionTests:test_1_basic_filter_catch ()' while running in aot-only mode. See https://docs.microsoft.com/xamarin/ios/internals/limitations for more information.
since BITCODE wasn't defined everywhere it needed to be.
* Instead of calculating a timeout based on the app size and an estimated
transfer speed, keep transferring as long as mlaunch outputs something to
stdout. The actual transfer speed can vary a lot, in particular if
installing to multiple nearby watches simultaneously.
* Improve progress text in the html report during installation to include more
statistics for the impatient.
By default xharness will clean the project after a successful test run (which
can be required if running many device tests without enormous amounts of disk
space).
However, sometimes this can be annoying, in particular if trying to re-run a
particular test manually.
So add an option in the UI to make cleaning optional.
The "Gizmo" and "Companion" are child elements, not attributes on the SimDevicePair.
Also replaced the custom Distinct() implementation with a comparer which can be used with standard LINQ.
The "Gizmo" and "Companion" are child elements, not attributes on the SimDevicePair.
Also replaced the custom Distinct() implementation with a comparer which can be used with standard LINQ.
Previously the queue at the right would move to the top (above the test list)
if a node with a lot of text in the test list was expanded. This is annoying
(since things would move out of view unexpectedly), so change the css so that
the queue on the right stays on the right no matter what.
This also required reordering the left and right divs in the html, which is
most of the diff.
This fixes an issue where we'd consume a thread pool thread until the launch
timeout if the app launched, but the test run never started (it crashed at
launch for instance).
Rework BCL test importation to generate projects that won't compile instead of
giving up completely in case of generation failure.
This means that xharness can still be used for non-BCL tests even if the
generation of the BCL tests fail.
* [tests] Use latest version of NUnit for test projects to get support for parallelized tests.
* [xharness] Automatically find the correct nunit-console executable for NUnit tests.
Alexander Köplinger
Vincent Dondain
Manuel de la Pena
Rolf Bjarne Kvinge
Bernhard Urban
Bernhard Urban