This commit improves the state of the BCL testing in the following ways:
1. Improve the device tets running. Less apps, faster results.
2. WatchOS apps are left as they were to ensure that we do not have deplouyment/run issues.
We now support the ignore files per assembly name to simplify the
tracking of the ignored tests. All
* [XHarness] Ensure we do a nuget restore on bcl imported tests. Fixes#5383
We need to ensure that a nuget restore is done, otherwise we might have
build issues with missing libs.
Fixes https://github.com/xamarin/xamarin-macios/issues/5383
Rework BCL test importation to generate projects that won't compile instead of
giving up completely in case of generation failure.
This means that xharness can still be used for non-BCL tests even if the
generation of the BCL tests fail.
- Existing binding projects embed the native libraries within the assembly as managed resource
- This does not scale well and has performance implications
- This PR creates a new property, NoBindingEmbedding which when true processes the building and consumption of binding projects differently.
- Existing binding projects are not affected, they will continue as is
- I've written a full XM test suite and ported a subset to iOS. Since iOS only supports checked in projects, and I didn't want to make the existing situation worse by adding more, I only wrote tests that could use the existing test projects.
-When we complete some form of msbuild testing reform, we'll revisit these tests.
- Remove two files in MyiOSFrameworkBinding that are not used (we use copies elsewhere)
- Remove unnecessary sleep and fix broken touch command
- Output failing test log to console instead of test output
- VSfM does not handle thousands of lines of test failure message well
- Add ability to generate binding projects with LinkWith
* [Xharness] Add a workaround to not build the xunit tests when not needed.
This is a workaround that does not use reflection to build the bcl test
projects. We need this until we have complete support of using mono as
an SDK.
Only build the BCL tests when we need them, because we can't build them when
we're running the device tests (because we don't have a built source tree in
that case).
Since the test assemblies are required when generating the BCL projects, this
means we also delay creating the BCL projects until we need them.
The ProtocolTest test is located in the bindings-test assembly, and it
requires certain conditional compilation defines to be set in order to behave
properly.
With this fix we correctly set these defines when cloning the bindings-test
project (in addition to when we set the defines for the main project, which is
either monotouch-test or linkall for this test).
Fixes https://github.com/xamarin/maccore/issues/655.
This fixes an issue where multiple copies of the GuiUnit project would have
the same output directory, causing random problems when building those
projects in parallel.
* [tests][xammac] Fix a few path calculations to not be hardcoded.
* [xharness] Clone XM projects.
Cloning projects before building them will also clone project references,
which will make it possible to build multiple projects in parallel, when those
projects have common project references.
Fixes https://github.com/xamarin/maccore/issues/624.
* [xharness] Fix XM project cloning
* Don't clone Classic projects (they're solutions, and xharness doesn't understand them).
* Clonee projects properly when cloning execution tasks.
* [xharness] GuiUnit needs specialized cloning.
The GuiUnit project uses relative paths to write to files outside the project
directory, which means that multiple GuiUnit project files may write to the
same location.
So special-case GuiUnit cloning to make those paths subdirectories of the
project's directory instead.
* [xharness] Process imported targets when cloning projects.
Also make the msbuild-mac's custom targets file independent of the location of
the project file by making all paths relative to the custom targets file.
* [xharness] Add support for running XI projects with different platform configurations.
* [xharness] Run dont link, link all and link sdk tests in both Debug and Release in the simulator. Fixes#53181.
Fixes https://bugzilla.xamarin.com/show_bug.cgi?id=53181.
* [tests] Fix dont link to always optimize C# code in release mode.
* [tests] No need to weak link GameController anymore in dont link.
This was removed long ago from the Debug configurations, so remove it from the
Release configuration as well.
* [xharness] Run monotouch-test both with and without the static registrar in the simulator.
This way we catch (some) problems with the static registrar without having to run on device.
* [xharness] Add comments to explain extra test configurations.
* [xharness] Fix how projects are cloned.
Fix how projects are cloned so that xharness doesn't end up confused when
multiple test tasks share the same build task.
Previously xharness would iterate over all test tasks and modifying their
build tasks, now it will create the build tasks correctly in the first place.
Change [Intermediate]OutputPath of Xamarin.Mac test projects to include the
project flavor, so that they don't build into (and run from) the same location
(which leads to random and incorrect behavior for at least one of the
flavors).
Fixes https://github.com/xamarin/maccore/issues/584
* [macos] Fix guiunit error on clean builds by depending on correct copy
- From a clean build making a BCL test would error due to the non-mobile guiunit not being built
- This was because the Makefile-mac.inc target was incorrect
- This was because xharness assumed that non variation based targets were always Modern
- However, BCL tests are Full, not Modern
* Code review change
* Swap to var to reduce diff
Rework how the two different xammac configurations are done, so that we don't
generate identical makefile targets for both variations (we'll generate a
single makefile target, and the configuration is selected using an environment
variable).
Fixes these makefile warnings:
Makefile-mac.inc:303: warning: overriding commands for target `build-mac-unified-xammac_tests'
Makefile-mac.inc:287: warning: ignoring old commands for target `build-mac-unified-xammac_tests'
Makefile-mac.inc:307: warning: overriding commands for target `clean-mac-unified-xammac_tests'
Makefile-mac.inc:291: warning: ignoring old commands for target `clean-mac-unified-xammac_tests'
Makefile-mac.inc:310: warning: overriding commands for target `exec-mac-unified-xammac_tests'
Makefile-mac.inc:294: warning: ignoring old commands for target `exec-mac-unified-xammac_tests'
Makefile-mac.inc:313: warning: overriding commands for target `run-mac-unified-xammac_tests'
Makefile-mac.inc:297: warning: ignoring old commands for target `run-mac-unified-xammac_tests'
https://bugzilla.xamarin.com/show_bug.cgi?id=56492
This affects introspection-ios/introspection-mac (which are now grouped in an
`introspection` test), and dontlink-mac (which now shows up in the `dont link`
tests).
* [ObjCRuntime] Remove XM limitation. Fixes#55857.
I can't remember why I added this limitation, and git history provides now
clues.
Everything compiles without the limitation, and the bug is fixed, so I can't
see a reason not to fix it.
https://bugzilla.xamarin.com/show_bug.cgi?id=55857
* [tests] Run xammac tests both in Debug and Release configurations.
* [tests] Convert mmp tests to a standard NUnit test library.
* [xharness] Add support for restoring nugets before building projects.
And restore nugets for tests-mac.sln before building mac test projects.
Make the top-level test name for equivalent tests between iOS and Mac have the
same name, so that these tests are grouped together.
For instance: we used to have a System-Mac tests and System tests (the latter
being iOS tests), now there are just System tests, and nested inside are both
the Mac and iOS System tests.
* Add a server mode, which launches a web server (and a web page) that can be
used to interactively run tests and view their results.
* Add support for running test assemblies in a today extension (generating a
new set of projects, similar to how we generate tvOS/watchOS projects based
on the iOS project, we now generate a today extension project in addition to
the tvOS and watchOS projects).
* Load all the different tests (and show them in the html report, although
they show up as 'ignored'), even for disabled/ignored tests. This makes
disabled/ignored tests more visible, and also makes it possible to actually
run them using the embedded web server.
* Add support for running tests on device. Tests will be executed on multiple
devices simulatenously (any connected devices will be used).
* [XM] Add 32-bit variants of apitest and introspection and fix bindings
- Previous to this we were only running 64-bit tests, which caused us
to make mistakes
* Fix cherry-pick to build
* Correctly skip jenkins XM45 when asked
* Fix logic
* Run 32-bit tests on Jenkins
* Fix Jenkins grouping by respecting -32 suffix
* More fixing of jenkins. This time make enum for 32-bit types
Manuel de la Pena
Alexander Köplinger
Rolf Bjarne Kvinge
Marek Safar
Chris Hamons
Martin Baulig