* [xharness] Change url for server mode.
* [xharness] Show links to previous test runs in html report in server mode.
This makes it much easier to see what failed in a previous test run.
This makes it possible to run tests that are marked as 'BuildOnly' to see if
they've been fixed or not.
There's already a 'build' button if only a build is desired.
Add support for running tests with the earliest possible simulator, and use it for introspection tests.
* Add support for running tests with the earliest possible simulator
(currently iOS 8.1, tvOS 9.1 and watchOS 2.0).
* Make the introspection tests run with the earliest possible simulator.
* Fix several binding issues (mostly missing availability attributes), and a
couple of issues in the introspection tests themselves.
Reference: https://github.com/xamarin/xamarin-macios/issues/3668.
The change allows to state the tests that have to be ran. ATM with these
changes, the vsts pipeline must add the following to the env vars:
* tvOS device pipelines: Must add run-tvos-tests to the labels.
* iOS device pipelines: Must add run-ios-tests to the labels.
This will ensure that only the tests for the devices are ran and if the
tests pass we get a green build with no unexpected skips.
We automatically create simulators when needed, but it won't work if the
simulator runtime isn't installed. So handle the case where a test might not
have a simulator to execute in correctly.
* [linker] Mark protocol interfaces when using the dynamic registrar.
Fixes this monotouch-test failure when using the dynamic registrar and the
linker at the same time:
[FAIL] RegistrarTest.TestProtocolRegistration : UIApplicationDelegate/17669
Expected: True
But was: False
* [tests] Adjust test after linker change.
All Xamarin.iOS apps will now link with QuickLook when using the dynamic
registrar, because NSUrl implements a QuickLook protocol:
fcac64ad6e/src/foundation.cs (L5445).
Adjust LinkAll_Frameworks accordingly, and add a new test that verifies that
the old behavior (not linking with QuickLook when linking all assemblies) is
still correct.
* Share much more marshalling code between the dynamic and static registrar
(by refactoring code into separate functions used by both).
* Fix an issue with multiple out/ref parameters in the dynamic registrar.
* Throw an error instead of silently ignoring out/ref parameter types we don't
completely understand in the dynamic registrar.
* Fix returning Class/SEL out/ref parameters in the static registrar.
* Fix returning NSString out/ref parameters in the dynamic registrar.
* Add/fix support for out/ref arrays everywhere.
* Fix support for the various supported out/ref parameter types in the
generator.
* Add lots of tests.
Fixes https://github.com/xamarin/xamarin-macios/issues/5171.
* [tests] Add sample tester.
Add a unit project that looks for iOS/macOS/tvOS sample projects in several
repositories, and builds them all.
* [tests][sampletester] Remove known issue which has now been fixed.
* [tests] Only run sample tests on CI in Azure Devops.
* Remove the possibility of automatically running the sample tests with
xharness (so the sample tests won't run on PR bots or internal bots when the
'run-all-tests' label is added). It's still possible to run the sample tests
manually from the xharness web UI.
* Automatically trigger the sample test run in Azure Devops if the
'run-sample-tests' label is applied to a PR (and that PR is executed on
internal Jenkins).
* Fix typo.
* Fix path.
* Verbose output to track down scheduling failure.
* Bump maccore to get improved debug spew.
Diff: f527c9c526..f89d74b165
* [tests][sampletester] Fix build for TodoWCF.
https://github.com/xamarin/xamarin-macios/pull/4884 introduced the logic of only building certain `RunTestTask`.
This was meant to disable iOS Extensions as part of a fix to https://github.com/xamarin/maccore/issues/1008.
However this didn't quite work and iOS extensions were still running (and failing).
The reason being that `BuildOnly` was set to a `RunDeviceTask` that's added to a list which is then given to `CreateTestVariations` which creates new instances of `RunDeviceTask`.
We now propagate `BuildOnly` to the new variation instance.
This commit improves the state of the BCL testing in the following ways:
1. Improve the device tets running. Less apps, faster results.
2. WatchOS apps are left as they were to ensure that we do not have deplouyment/run issues.
We now support the ignore files per assembly name to simplify the
tracking of the ignored tests. All
* [XHarness] Ensure we do a nuget restore on bcl imported tests. Fixes#5383
We need to ensure that a nuget restore is done, otherwise we might have
build issues with missing libs.
Fixes https://github.com/xamarin/xamarin-macios/issues/5383
Most of the tests are comming from the downloaded SDK, but some of them
have to reman because they are running/passing more tests. This
differences are due to how the tests that come from mono are written
(missing resources etc..)
Please take a look at
https://devdiv.visualstudio.com/DevDiv/_workitems/edit/794210/ for the
list of bcl tests and test number differences.
It's harder to forget setting the define if it's already set by default.
Fixes the mini tests on watchOS device:
[FAIL] JitTests.Exceptions : System.Reflection.TargetInvocationException : Exception has been thrown by the target of an invocation.
----> System.ExecutionEngineException : Attempting to JIT compile method 'ExceptionTests:test_1_basic_filter_catch ()' while running in aot-only mode. See https://docs.microsoft.com/xamarin/ios/internals/limitations for more information.
since BITCODE wasn't defined everywhere it needed to be.
* Instead of calculating a timeout based on the app size and an estimated
transfer speed, keep transferring as long as mlaunch outputs something to
stdout. The actual transfer speed can vary a lot, in particular if
installing to multiple nearby watches simultaneously.
* Improve progress text in the html report during installation to include more
statistics for the impatient.
Rolf Bjarne Kvinge
Manuel de la Pena
Alexander Köplinger
Vincent Dondain