This change adds support to execute the tests provided by mono as assemblies.This includes:
1. App generation to run the bcl tests on modern and full.
2. Needed workaround to delay the compilation of the assemblies until we have them from the SDK.
3. All failing tests are ignored.
* [Harness] Reduce the noise in the make files.
The Makefile.cs code generates a lot of noise becuase it tries to build
a make rule for the new BCL tests, these rules do not work and can be
ignored.
Fixes issue: https://github.com/xamarin/maccore/issues/1156
* xibuild: New wrapper tool to run msbuild or managed executables
MSBuild supports fallback paths for projects imported using
`$(MSBuildExtensionsPath)`, but these must be specified explicitly in
the app.config of the main executable. There was a PR to allow use of
properties for this in the app.config, but that was not accepted
upstream.
This is required for being able to:
1. build projects with msbuild against the in-tree XI/XM build output
2. and to run nunit tests against the same.
For this we introduce a new tool, `xibuild`, based on XA's `xabuild`.
This supports the fallback paths to be specified via the environment variable
`MSBuildExtensionsPathFallbackPathsOverride`[1].
It essentially operates in 3 modes:
1. `xibuild -c /path/to/foo.exe`
Generates /path/to/foo.exe.config with the fallback paths inserted into that.
2. `xibuild -- /v:diag /path/to/project.csproj`
Runs msbuild with the arguments after `--` with a custom app.config based on
`MSBuild.dll.config`, with the fallback paths correctly inserted.
This is in a temporary file and the original config file is not touched.
3. `xibuild -t -- /path/to/managed_tool.exe args`
Generates `/path/to/managed_tool.exe.config` based on `MSBuild.dll.config` with
the fallback paths inserted, and runs `managed_tool.exe` with the arguments.
The default is to overwrite the config file.
But there is also a switch to merge it with an existing config file.
--
1. Value of the environment variable $MSBuildExtensionsPathFallbackPathsOverride
is prepended to any existing list of search paths in `MSBuild.dll.config`, IOW,
it takes precedence. So, the order of lookup becomes:
- Value of the property `$(MSBuildExtensionsPath)`
- Value of the environment variable `$MSBuildExtensionsPathFallbackPathsOverride`
- /Library/Frameworks/Mono.framework/External/xbuild on macOS
* Integrate use of `xibuild` with the tests
Update all uses of `msbuild` and invocations of tools like nunit that
might depend on using the in-tree builds to use `xibuild`.
* xibuild: Move help descriptions to OptionSet itself.
Make sure we don't run 'nuget restore' for test-mac.sln multiple times in
parallel by creating a separate make target that runs 'nuget restore', and
making every build-mac-* target depend on that target.
Fixes https://github.com/xamarin/maccore/issues/1122.
* [tests] Point MSBuild to the right Xamarin.Mac location when building packaged Xamarin.Mac tests. Fixes maccore#1115.
Fixes https://github.com/xamarin/maccore/issues/1115.
* [xharness] Ensure all makefile targets set the proper environment variables.
- And map xbuild properties to msbuild ones for fallback paths
`XBUILD_FRAMEWORK_FOLDERS_PATH -> TargetFrameworkFallbackSearchPaths`
`MSBuildExtensionsPath -> MSBuildExtensionsPathFallbackPathsOverride`
Note:
Earlier with xbuild, the order of lookup for (example)
`MSBuildExtensionsPath` was:
1. The value of $(MSBuildExtensionsPath), which we were setting to
the in-tree path
2. /Library/Frameworks/Mono.framework/External/xbuild on osx
3. $prefix/lib/mono/xbuild (default location)
And with the above changes, it will be:
1. The value of $(MSBuildExtensionsPath), which we are no longer
setting, so the default path : $prefix/lib/mono/xbuild
2. The in-tree path, via $(MSBuildExtensionsPathFallbackPathsOverride)
3. /Library/Frameworks/Mono.framework/External/xbuild on osx
Since, XI/XM targets are used via fallback path
`/Library/Frameworks/Mono.framework/External/xbuild`, the default
location doesn't matter. And the order of the remaining two remains the
same.
The same thing applies to the target frameworks also.
These targets will already be properly escaped (using backslash), so adding
quotes will cause make to treat the backslashes literally (and break).
Fixes an issue with package-tests:
$ make package-tests
[...]
build-mac-classic-dont\ link failed
build-mac-unified-dont\ link failed
build-mac-unifiedXM45-dont\ link failed
build-mac-unified32-dont\ link failed
build-mac-unifiedXM4532-dont\ link failed
build-mac-unified-link\ all failed
build-mac-unified-link\ sdk failed
make[4]: *** [build-mac] Error 1
make[3]: *** [mac-test-package.zip] Error 2
make[2]: *** [package-tests] Error 2
* [tests] Add 'link all' test for XM.
* [tests] Add 'link sdk' test for XM.
* [tests] Move dontlink-mac tests to linker-mac directory to have the same directory layout as linker-ios.
* [tests] Convert mmp tests to a standard NUnit test library.
* [xharness] Add support for restoring nugets before building projects.
And restore nugets for tests-mac.sln before building mac test projects.
* Use Visual Studio instead of Xamarin Studio.
* VS doesn't have mdtool, it has vstool.
Also there's no need to manually invoke the mdtool.exe executable anymore
(which we did because the mdtool executable had a min macOS version of 10.9,
and we used to build tests on older macOS versions [1]), since now we only run
tests on older macOS versions, we don't build those tests there.
[1] a1932b0ccd
* Add a server mode, which launches a web server (and a web page) that can be
used to interactively run tests and view their results.
* Add support for running test assemblies in a today extension (generating a
new set of projects, similar to how we generate tvOS/watchOS projects based
on the iOS project, we now generate a today extension project in addition to
the tvOS and watchOS projects).
* Load all the different tests (and show them in the html report, although
they show up as 'ignored'), even for disabled/ignored tests. This makes
disabled/ignored tests more visible, and also makes it possible to actually
run them using the embedded web server.
* Add support for running tests on device. Tests will be executed on multiple
devices simulatenously (any connected devices will be used).
* [tests] Implement filtering based on test name to make it possible to run a subset of tests manually.
* [xharness] Add support for passing environment variables to test apps.
* [xharness] Split the mscorlib/System test run for watchOS into multiple executions.
When running on a watch, the complete set of mscorlib and System tests use too
much memory, so they end up crashing the app.
So instead don't run all the tests in the same test run, but
instead split them up.
This splits the System tests in 5 different sets, based on the first
character of the test class name.
The mscorlib tests are split in 26 different sets, one for each
character (A-Z).
Alexander Köplinger
Manuel de la Pena
Ankit Jain
Rolf Bjarne Kvinge
Chris Hamons
Sebastien Pouliot