Usually it's when working on beta Xcodes we make mistakes that only show up
when running on older simulators (and devices):
* Missing/wrong availability attributes.
* Tests for new API that don't check the OS if that API is available.
So automatically enable the tests on older simulators for PR builds when using
a beta Xcode.
* [tests] Remove 32-bit Xamarin.Mac tests (both Classic and Unified).
We're removing support for 32-bit Xamarin.Mac apps (#6300), which means we
don't need to run the tests anymore.
This part of the implementation for #6300, I'm starting with the tests because
some of our XM/Classic tests are failing and making the bots unnecessarily red
(since we'll remove XM/Classic support anyway). Also CI will be faster if we
don't run these tests.
* [mmptest] Fix build.
* [tests] Fix build for mono-native-mac.csproj.
This commit improves the state of the BCL testing in the following ways:
1. Improve the device tets running. Less apps, faster results.
2. WatchOS apps are left as they were to ensure that we do not have deplouyment/run issues.
We now support the ignore files per assembly name to simplify the
tracking of the ignored tests. All
Most of the tests are comming from the downloaded SDK, but some of them
have to reman because they are running/passing more tests. This
differences are due to how the tests that come from mono are written
(missing resources etc..)
Please take a look at
https://devdiv.visualstudio.com/DevDiv/_workitems/edit/794210/ for the
list of bcl tests and test number differences.
* Instead of calculating a timeout based on the app size and an estimated
transfer speed, keep transferring as long as mlaunch outputs something to
stdout. The actual transfer speed can vary a lot, in particular if
installing to multiple nearby watches simultaneously.
* Improve progress text in the html report during installation to include more
statistics for the impatient.
Rework BCL test importation to generate projects that won't compile instead of
giving up completely in case of generation failure.
This means that xharness can still be used for non-BCL tests even if the
generation of the BCL tests fail.
Jenkins has a limitation where you can't mark a step a failure, it has to
*fail* to be reported as such.
This means that running multiple tests, and reporting a failure if any of
those tests fail is not possible.
We run into this with the normal test run + the docs tests; where we currently
don't show a red result in the UI if any of those fail.
So incorporate the test-docs step into xharness, so that we only have one
thing to in the Jenkinsfile, which makes it possible for us to fail the test
run step properly.
This also required a few upgrades to xharness to get more info for pull
requests, since the logic to enable the docs tests is a bit more complicated
than anything else we have (if the current branch (or the target branch for a
PR) is 'master' AND xamarin mode is enabled).
We now download the ios Mono sdk which means that there is no need for
us to build the test assemblies on iOS, WatchOS and tvOS BUT we need to
do so for the Mac tests.
The change includes:
1. Use fo the downloaded test assemblies.
2. Removal of the dependency of building them in the tests.
3. Move back to using reflection for the project generation.
4. Remove the cached project details added in the workaround.
This change adds support to execute the tests provided by mono as assemblies.This includes:
1. App generation to run the bcl tests on modern and full.
2. Needed workaround to delay the compilation of the assemblies until we have them from the SDK.
3. All failing tests are ignored.
* [Xharness] Add a workaround to not build the xunit tests when not needed.
This is a workaround that does not use reflection to build the bcl test
projects. We need this until we have complete support of using mono as
an SDK.
* Use a full path to xibuild.
Use a full path to xibuild everywhere, since it's easier than making sure PATH
is correct every time we want to invoke xibuild.
Also remove the xbuild-in-place script, it's not used anymore.
* Fix xibuild path lookup.
* [xammac_tests] Remove unneeded csproj changes.
Only build the BCL tests when we need them, because we can't build them when
we're running the device tests (because we don't have a built source tree in
that case).
Since the test assemblies are required when generating the BCL projects, this
means we also delay creating the BCL projects until we need them.
Mono know provides a set of testing assemblies that contain the tests to be ran in each of the supported platforms. This commit adds the following:
1. Two different runners that can be used to execute NUnit and xUnit tests.
2. The necessary glue to get xharness to run the new tests and report the results.
3. A new test that will ensure that if mono adds new tests, xharness will report failures if the tests are not run or not ignored.
Unce upon a time we used a separate mono submodule for watchOS support, to make
development of watchOS support easier (we referenced mono/master, to avoid
backporting fixes for watchOS support through various release branches in
mono).
This only worked until our watchOS support became stable, since then we had to
start using a stable version of mono for watchOS support.
This means that our build support for using a separate mono clone for watchOS
support is no longer needed; and in any case it's broken because of build
changes done later.
* [jenkins] Only XM apps with variations are Classic/32-bit apps, so adjust ignore logic accordingly. Fixes maccore issue 884.
Fixes https://github.com/xamarin/maccore/issues/884.
* [mmp] Fix passing -stdlib=libc++ to clang.
* [tests] Fix 32-bit XM issues.
* [xharness] Add support for building 32-bit XM apps by using Xcode 9.4.
* [xharness] Since xharness can now build 32-bit mac apps, enable them by default.
* Remove debug code.
* Remove unused variable.
* Put a binary copy of mlaunch in macios-binaries instead of in Azure. Fixes#3316.
If the Xamarin build is enabled, then build mlaunch from source, otherwise get it from macios-binaries.
Fixes#3316.
* Bump macios-binaries.
Commit list for xamarin/macios-binaries:
* xamarin/macios-binaries@69a9088 Bump mlaunch to xamarin/maccore@4505cd6f02 (#8)
Diff: e1e8bdf7a8...69a90882a0
* make msbuild/ after tools/.
Build msbuild/ after tools/, so that mlaunch builds when building from source,
since its build needs the MSBuild logic installed by msbuild/.
* Bump macios-binaries to get Xcode 9.3-capable mlaunch.
* Put a binary copy of mlaunch in macios-binaries instead of in Azure. Fixes#3316.
If the Xamarin build is enabled, then build mlaunch from source, otherwise get it from macios-binaries.
Fixes#3316.
* Bump macios-binaries.
Commit list for xamarin/macios-binaries:
* xamarin/macios-binaries@69a9088 Bump mlaunch to xamarin/maccore@4505cd6f02 (#8)
Diff: e1e8bdf7a8...69a90882a0
* [tests] Move linker tests to match introspection directory layout.
Move linker tests to match introspectio directory layout: tests/linker/ios and
tests/linker/mac instead of tests/linker-ios and tests/linker-mac.
This creates a logical place for shared linker files (tests/linker).
* [tests] Fix path to GuiUnit_NET_4_5.csproj in sln as well.
This is the path used to find referenced projects when building Classic.
Fixes this build problem:
warning: Referenced project 'GuiUnit_NET_4_5' not found in the solution.
/Users/builder/jenkins/workspace/xamarin-macios-pr-builder/tests/common/mac/MacTestMain.cs(11,7) : error CS0246: The type or namespace name `GuiUnit' could not be found. Are you missing an assembly reference?
/Users/builder/jenkins/workspace/xamarin-macios-pr-builder/tests/common/mac/MacTestMain.cs(64,42) : error CS0246: The type or namespace name `IMainLoopIntegration' could not be found. Are you missing an assembly reference?
/Users/builder/jenkins/workspace/xamarin-macios-pr-builder/tests/common/mac/MacTestMain.cs(75,34) : error CS0246: The type or namespace name `InvokerHelper' could not be found. Are you missing an assembly reference?
* [xharness] Add support for running XI projects with different platform configurations.
* [xharness] Run dont link, link all and link sdk tests in both Debug and Release in the simulator. Fixes#53181.
Fixes https://bugzilla.xamarin.com/show_bug.cgi?id=53181.
* [tests] Fix dont link to always optimize C# code in release mode.
* [tests] No need to weak link GameController anymore in dont link.
This was removed long ago from the Debug configurations, so remove it from the
Release configuration as well.
Rolf Bjarne Kvinge
Manuel de la Pena
Alexander Köplinger
Marek Safar
Martin Baulig
Chris Hamons
Vincent Dondain
Sebastien Pouliot