Move all the extension methods to a class. After this refactor, we will
be able to DI the manager in the other classes and assert that the
processes are called with the correct parameters without the need of
launching them.
Also added tests for the manager. We create a dummy console app that
will be executed by the tests. The console app has a number of
parameters that will be used to ensure that the new process behaves as
we want:
- Use the passed exit code.
- Create child proecesses if needed.
- Sleep to force a timeout.
- Writer messages to stdout and stderr.
Our tests call the dummy app and ensures that the results match the
behaviour expected by the dummy app.
Co-authored-by: Manuel de la Pena <mandel@microsoft.com>
Co-authored-by: Premek Vysoky <prvysoky@microsoft.com>
Co-authored-by: Rolf Bjarne Kvinge <rolf@xamarin.com>
* [Harness] Refactor process management to be testable.
Move all the extension methods to a class. After this refactor, we will
be able to DI the manager in the other classes and assert that the
processes are called with the correct parameters without the need of
launching them.
Also added tests for the manager. We create a dummy console app that
will be executed by the tests. The console app has a number of
parameters that will be used to ensure that the new process behaves as
we want:
- Use the passed exit code.
- Create child proecesses if needed.
- Sleep to force a timeout.
- Writer messages to stdout and stderr.
Our tests call the dummy app and ensures that the results match the
behaviour expected by the dummy app.
* Apply suggestions from code review
Co-Authored-By: Přemek Vysoký <premek.vysoky@microsoft.com>
* Move out utils into a separate namespace
* Move Cache to the test project
* Fix namespaces after merge
* Remove unneeded code
* Move Target files
* Sort csproj
* Refactor Targets
* Rename Cache to TempDirectory
* Fix using
* Move ProjectFileExtensions
* Remove dead code
* Move Extensions
* Add empty StringUtilsTests
* Add StringUtils tests
* Revert refactorings
* Update tests/xharness/Utilities/StringUtils.cs
Co-Authored-By: Rolf Bjarne Kvinge <rolf@xamarin.com>
* Update tests/xharness/Utilities/StringUtils.cs
Co-Authored-By: Rolf Bjarne Kvinge <rolf@xamarin.com>
* Add better formatarguments test
* Update tests/xharness/Utilities/StringUtils.cs
Co-Authored-By: Rolf Bjarne Kvinge <rolf@xamarin.com>
Co-authored-by: Manuel de la Pena <mandel@microsoft.com>
Co-authored-by: Premek Vysoky <prvysoky@microsoft.com>
Co-authored-by: Rolf Bjarne Kvinge <rolf@xamarin.com>
Move to use interfaces, that will let us later add tests that will
verify that all the correct logging is performed. As an example, added a
test for XmlResultParser that ensures that the failures are correctly
generated. The test uses Moq to pass the different paths to be used and
later be able to verify the wirtten xml.
Unify the harness properties to just look at InCI and remove all the
other ones. There is no real need to have differences between jenkins
and VSTS and Wrench is gone.
Bump mono to get the new splited test dlls and add them to be ran in
xharness. Special logic is used for mscorlib so we make sure that all
the 'parts' of the test dll do have the same configurations.
Co-Authored-By: Waleed Chaudhry <54864665+wachaudh@users.noreply.github.com>
* macOS 10.15 starts putting up permission dialogs we can't automatically
dismiss anymore, so start honoring the 'IncludeSystemPermissionTests' option
for macOS tests.
* Improve the 'IncludeSystemPermissionTests' option to have three states: if
set (either true or false), that takes precedence, but if not set, we now
don't run any tests that require permission dialogs on macOS or on device if
we're running in CI. Tests executed locally will still put up dialogs, both
on macOS and on device.
* This needed a few changes to the html report, since the
'IncludeSystemPermissionTests' is exposed in the UI and the code didn't
handle the three different states.
* Update a few tests to check for permission to the contacts.
Fixes https://github.com/xamarin/maccore/issues/1856.
This boils down to the makefile-generation code having the information it
needs (and that information being correct).
This fixes running of tests on other macOS bots (older/newer), because now we
can build the test package again.
* Bump VSMac to 8.1.0.2742 to fix msbuild issues (#6279)
* Bump VSMac to 8.1.0.2742 to fix msbuild issues
This is required to get the support for the msbuild `ToolsVersion`
change from `15.0` to `Current`.
* [tests][msbuild] Fix Binding resources test with updated msbuild
Test failure with updated msbuild and vsmac 8.1:
```
Xamarin.iOS.Tasks.NativeReferencesNoEmbedding("iPhone").ShouldNotUnnecessarilyRebuildBindingProject(True)
Binding project build did not create package?
Expected: True
But was: False
at Xamarin.iOS.Tasks.NativeReferencesNoEmbedding.ShouldNotUnnecessarilyRebuildBindingProject (System.Boolean framework) [0x000a0] in <74b8f7d8a53e40109916d305bb4d7403>:0
at (wrapper managed-to-native) System.Reflection.RuntimeMethodInfo.InternalInvoke(System.Reflection.RuntimeMethodInfo,object,object[],System.Exception&)
at System.Reflection.RuntimeMethodInfo.Invoke (System.Object obj, System.Reflection.BindingFlags invokeAttr, System.Reflection.Binder binder, System.Object[] parameters, System.Globalization.CultureInfo cul
ture) [0x0006a] in <0519fa732e8845b6a809ce9180f541db>:0
```
The test builds the project multiple times. Before the 3rd build, the project
file's timestamp is updated and expects that the binding package will be
rebuilt. But it is not, because the target `_CreateBindingResourcePackage`
doesn't depend on that project file. So, add that to the target inputs.
* [nuget] Use xibuild to run nuget
Fix errors seen during `nuget restore` for tests:
```
Users/builder/jenkins/workspace/xamarin-macios-pr-builder/tests/xammac_tests/xammac_tests.csproj(213,3): error MSB4024: The imported project file "/Library/Frameworks/Mono.framework/External/xbuild/Xamarin/Mac/Xamarin.Mac.CSharp.targets" could not be loaded. Could not find file "/Library/Frameworks/Mono.framework/External/xbuild/Xamarin/Mac/Xamarin.Mac.CSharp.targets"
```
* [xibuild] Fix incorrect mscorlib.dll being used (#6068)
* [xibuild] Fix incorrect mscorlib.dll being used
The `GuiUnit_NET_4_5` project, when built with `xibuild` uses the wrong `mscorlib.dll`.
From https://github.com/xamarin/xamarin-macios/issues/5760#issuecomment-472457202 :
```
- mscorlib.dll is being used from mono/4.5 and the other system assemblies are from mono/4.5-api
- GuiNet* project is built with xibuild
What is happening here is:
xibuild sets[1] `SetToolsetProperty ("TargetFrameworkRootPath", FrameworksDirectory + Path.DirectorySeparatorChar);`
which points to `/Library/Frameworks/Mono.framework/Versions/Current/lib/mono/xbuild-frameworks`.
This causes $(FrameworkPathOverride) to be set[2] to `/Library/Frameworks/Mono.framework/Versions/Current/lib/mono/xbuild-frameworks/.NETFramework/v4.5`,
but that doesn't have a mscorlib.dll, so it gets reset[3] to /Library/Frameworks/Mono.framework/Versions/5.22.0/lib/mono/4.5/.
If we don't set TargetFrameworkRoothPath, then we get `FrameworkPathOverride = /Library/Frameworks/Mono.framework/Versions/5.22.0/lib/mono/4.5-api`,
causing `_ExplicitReference=/Library/Frameworks/Mono.framework/Versions/5.22.0/lib/mono/4.5-api/mscorlib.dll`(correct one) to be used.
```
Fixes https://github.com/xamarin/xamarin-macios/issues/5760
1. https://github.com/xamarin/xamarin-macios/blob/master/tools/xibuild/Main.cs#L209
2. https://github.com/mono/msbuild/blob/xplat-master/src/Tasks/Microsoft.Common.CurrentVersion.targets#L79
3. https://github.com/mono/msbuild/blob/xplat-master/src/Tasks/Microsoft.Common.CurrentVersion.targets#L84
* Revert "Workaround https://github.com/xamarin/xamarin-macios/issues/5760 in generator csproj"
This reverts commit 9bd927bb7f.
The previous commit for xibuild removes the need for this.
* [xibuild] Handle "incorrectly" cased msbuild property names (#6202)
msbuild property names are case insensitive. While generating the custom
app.config, in `SetToolsetProperty(..)` we try to update the property if
it already exists. But the name lookup was case sensitive, thus causing
the lookup to fail, resulting in two entries for the same property name
differing only in case. Eg. `MSBuildSDKsPath` vs `MSBuildSdksPath`.
Fixed to ignore case.
Fixes https://github.com/mono/mono/issues/14765 .
* Bump VSMac to 8.1.0.2742 to fix msbuild issues
This is required to get the support for the msbuild `ToolsVersion`
change from `15.0` to `Current`.
* [tests][msbuild] Fix Binding resources test with updated msbuild
Test failure with updated msbuild and vsmac 8.1:
```
Xamarin.iOS.Tasks.NativeReferencesNoEmbedding("iPhone").ShouldNotUnnecessarilyRebuildBindingProject(True)
Binding project build did not create package?
Expected: True
But was: False
at Xamarin.iOS.Tasks.NativeReferencesNoEmbedding.ShouldNotUnnecessarilyRebuildBindingProject (System.Boolean framework) [0x000a0] in <74b8f7d8a53e40109916d305bb4d7403>:0
at (wrapper managed-to-native) System.Reflection.RuntimeMethodInfo.InternalInvoke(System.Reflection.RuntimeMethodInfo,object,object[],System.Exception&)
at System.Reflection.RuntimeMethodInfo.Invoke (System.Object obj, System.Reflection.BindingFlags invokeAttr, System.Reflection.Binder binder, System.Object[] parameters, System.Globalization.CultureInfo cul
ture) [0x0006a] in <0519fa732e8845b6a809ce9180f541db>:0
```
The test builds the project multiple times. Before the 3rd build, the project
file's timestamp is updated and expects that the binding package will be
rebuilt. But it is not, because the target `_CreateBindingResourcePackage`
doesn't depend on that project file. So, add that to the target inputs.
* [nuget] Use xibuild to run nuget
Fix errors seen during `nuget restore` for tests:
```
Users/builder/jenkins/workspace/xamarin-macios-pr-builder/tests/xammac_tests/xammac_tests.csproj(213,3): error MSB4024: The imported project file "/Library/Frameworks/Mono.framework/External/xbuild/Xamarin/Mac/Xamarin.Mac.CSharp.targets" could not be loaded. Could not find file "/Library/Frameworks/Mono.framework/External/xbuild/Xamarin/Mac/Xamarin.Mac.CSharp.targets"
```
* [tests] Remove 32-bit Xamarin.Mac tests (both Classic and Unified).
We're removing support for 32-bit Xamarin.Mac apps (#6300), which means we
don't need to run the tests anymore.
This part of the implementation for #6300, I'm starting with the tests because
some of our XM/Classic tests are failing and making the bots unnecessarily red
(since we'll remove XM/Classic support anyway). Also CI will be faster if we
don't run these tests.
* [mmptest] Fix build.
* [tests] Fix build for mono-native-mac.csproj.
The change is a pre step to fix https://github.com/xamarin/xamarin-macios/issues/6162
We are removing the not longer child directory bcl-test/BCLTests since
there is not longer any reason for it to exist. The changes make sure
that the tests work as expected in the new directory.
Next step, move all the test apps to their own directory.
This change adds support to execute the tests provided by mono as assemblies.This includes:
1. App generation to run the bcl tests on modern and full.
2. Needed workaround to delay the compilation of the assemblies until we have them from the SDK.
3. All failing tests are ignored.
* [Harness] Reduce the noise in the make files.
The Makefile.cs code generates a lot of noise becuase it tries to build
a make rule for the new BCL tests, these rules do not work and can be
ignored.
Fixes issue: https://github.com/xamarin/maccore/issues/1156
* xibuild: New wrapper tool to run msbuild or managed executables
MSBuild supports fallback paths for projects imported using
`$(MSBuildExtensionsPath)`, but these must be specified explicitly in
the app.config of the main executable. There was a PR to allow use of
properties for this in the app.config, but that was not accepted
upstream.
This is required for being able to:
1. build projects with msbuild against the in-tree XI/XM build output
2. and to run nunit tests against the same.
For this we introduce a new tool, `xibuild`, based on XA's `xabuild`.
This supports the fallback paths to be specified via the environment variable
`MSBuildExtensionsPathFallbackPathsOverride`[1].
It essentially operates in 3 modes:
1. `xibuild -c /path/to/foo.exe`
Generates /path/to/foo.exe.config with the fallback paths inserted into that.
2. `xibuild -- /v:diag /path/to/project.csproj`
Runs msbuild with the arguments after `--` with a custom app.config based on
`MSBuild.dll.config`, with the fallback paths correctly inserted.
This is in a temporary file and the original config file is not touched.
3. `xibuild -t -- /path/to/managed_tool.exe args`
Generates `/path/to/managed_tool.exe.config` based on `MSBuild.dll.config` with
the fallback paths inserted, and runs `managed_tool.exe` with the arguments.
The default is to overwrite the config file.
But there is also a switch to merge it with an existing config file.
--
1. Value of the environment variable $MSBuildExtensionsPathFallbackPathsOverride
is prepended to any existing list of search paths in `MSBuild.dll.config`, IOW,
it takes precedence. So, the order of lookup becomes:
- Value of the property `$(MSBuildExtensionsPath)`
- Value of the environment variable `$MSBuildExtensionsPathFallbackPathsOverride`
- /Library/Frameworks/Mono.framework/External/xbuild on macOS
* Integrate use of `xibuild` with the tests
Update all uses of `msbuild` and invocations of tools like nunit that
might depend on using the in-tree builds to use `xibuild`.
* xibuild: Move help descriptions to OptionSet itself.
Make sure we don't run 'nuget restore' for test-mac.sln multiple times in
parallel by creating a separate make target that runs 'nuget restore', and
making every build-mac-* target depend on that target.
Fixes https://github.com/xamarin/maccore/issues/1122.
* [tests] Point MSBuild to the right Xamarin.Mac location when building packaged Xamarin.Mac tests. Fixes maccore#1115.
Fixes https://github.com/xamarin/maccore/issues/1115.
* [xharness] Ensure all makefile targets set the proper environment variables.
- And map xbuild properties to msbuild ones for fallback paths
`XBUILD_FRAMEWORK_FOLDERS_PATH -> TargetFrameworkFallbackSearchPaths`
`MSBuildExtensionsPath -> MSBuildExtensionsPathFallbackPathsOverride`
Note:
Earlier with xbuild, the order of lookup for (example)
`MSBuildExtensionsPath` was:
1. The value of $(MSBuildExtensionsPath), which we were setting to
the in-tree path
2. /Library/Frameworks/Mono.framework/External/xbuild on osx
3. $prefix/lib/mono/xbuild (default location)
And with the above changes, it will be:
1. The value of $(MSBuildExtensionsPath), which we are no longer
setting, so the default path : $prefix/lib/mono/xbuild
2. The in-tree path, via $(MSBuildExtensionsPathFallbackPathsOverride)
3. /Library/Frameworks/Mono.framework/External/xbuild on osx
Since, XI/XM targets are used via fallback path
`/Library/Frameworks/Mono.framework/External/xbuild`, the default
location doesn't matter. And the order of the remaining two remains the
same.
The same thing applies to the target frameworks also.
These targets will already be properly escaped (using backslash), so adding
quotes will cause make to treat the backslashes literally (and break).
Fixes an issue with package-tests:
$ make package-tests
[...]
build-mac-classic-dont\ link failed
build-mac-unified-dont\ link failed
build-mac-unifiedXM45-dont\ link failed
build-mac-unified32-dont\ link failed
build-mac-unifiedXM4532-dont\ link failed
build-mac-unified-link\ all failed
build-mac-unified-link\ sdk failed
make[4]: *** [build-mac] Error 1
make[3]: *** [mac-test-package.zip] Error 2
make[2]: *** [package-tests] Error 2
* [tests] Add 'link all' test for XM.
* [tests] Add 'link sdk' test for XM.
* [tests] Move dontlink-mac tests to linker-mac directory to have the same directory layout as linker-ios.
* [tests] Convert mmp tests to a standard NUnit test library.
* [xharness] Add support for restoring nugets before building projects.
And restore nugets for tests-mac.sln before building mac test projects.
* Use Visual Studio instead of Xamarin Studio.
* VS doesn't have mdtool, it has vstool.
Also there's no need to manually invoke the mdtool.exe executable anymore
(which we did because the mdtool executable had a min macOS version of 10.9,
and we used to build tests on older macOS versions [1]), since now we only run
tests on older macOS versions, we don't build those tests there.
[1] a1932b0ccd
* Add a server mode, which launches a web server (and a web page) that can be
used to interactively run tests and view their results.
* Add support for running test assemblies in a today extension (generating a
new set of projects, similar to how we generate tvOS/watchOS projects based
on the iOS project, we now generate a today extension project in addition to
the tvOS and watchOS projects).
* Load all the different tests (and show them in the html report, although
they show up as 'ignored'), even for disabled/ignored tests. This makes
disabled/ignored tests more visible, and also makes it possible to actually
run them using the embedded web server.
* Add support for running tests on device. Tests will be executed on multiple
devices simulatenously (any connected devices will be used).
* [tests] Implement filtering based on test name to make it possible to run a subset of tests manually.
* [xharness] Add support for passing environment variables to test apps.
* [xharness] Split the mscorlib/System test run for watchOS into multiple executions.
When running on a watch, the complete set of mscorlib and System tests use too
much memory, so they end up crashing the app.
So instead don't run all the tests in the same test run, but
instead split them up.
This splits the System tests in 5 different sets, based on the first
character of the test class name.
The mscorlib tests are split in 26 different sets, one for each
character (A-Z).
Rolf Bjarne Kvinge
Přemek Vysoký
Manuel de la Pena
Ankit Jain
Alexander Köplinger
Chris Hamons
Sebastien Pouliot