One of the reason why we lost the TestReport was because the xml added
from the new code did not use the same name for the log as the expected
one (https://github.com/xamarin/xamarin-macios/pull/7835/files#diff-03773861bcef485bf343300f31b60b0eR374).
If we are going to be using the description as a way to decide what is
going to be done with it, lets use static vars so that we do not have
logs added with the wrong name.
Add code to write the TestReport when we have xunit results. Also
refactored code for the NUnit format to not use LINQ, it will use less
memory and in the future we can move to async (not now since it might
raise other problems).
fixes: https://github.com/xamarin/xamarin-macios/issues/7826
Only one to start... it's been discussed before but we generally
found other ways to do them. Let's continue to pick the best place
but we now have more options :)
Use the xml parsing helper methods to decide if the xml can be parsed
and if we will be able to generate the report. That way we avoid an
exception that makes the CI noise.
We ping the tcp listener to know that we have a tcp connection, that is
written in the xml logs, which means that parsing will not work. Ignore
the ping, parse xml, and make sure that the xml that will be consume by
vsts is valid.
This PR also fixes https://github.com/xamarin/maccore/issues/827 which does not longer happen.
* Acquire permission for the microphone (the microphone permission dialog is
not a blocking dialog, which is why things have worked so far).
* Set the last_modified field in the TCC database to now instead of 1970.
Apparently some permissions time out (kTCCServiceMediaLibrary), which means
that the last_modified field is important to get right.
Looks like we have issues with the internet sharing in the VSTS bots,
this means that now that we always try to parse the XML on CI, we get an
exception, catch it and do not show the results.
The workaround simply tries to read the xml, if possible, we will parse
it, else deal with the text only log. The fix will show the results, but
it is a workaround for a configuration issue in the CI.
Unify the harness properties to just look at InCI and remove all the
other ones. There is no real need to have differences between jenkins
and VSTS and Wrench is gone.
We used to test only on Jenkins, and if the build version had jenkins on
it, rather than doing so, just check if we are in the CI by looking if
BUILD_REVISION is present in the env.
* Bump Xamarin.MacDev.
New commits in xamarin/Xamarin.MacDev:
* xamarin/Xamarin.MacDev@210c664 Adds net451 to Xamarin.MacDev.csproj
* xamarin/Xamarin.MacDev@64db365 [winios] Changes provisioning profiles default path
* xamarin/Xamarin.MacDev@d34430a Switch to short-form projects and build for both net461 and netstandard2.0. (#68)
Diff: 0f578f51e6..210c664e56
* [msbuild] Update to latest Mono.Cecil.
The older version doesn't support netstandard2.0.
No code changes were required.
* [msbuild] Remove unused usings.
* [msbuild] Make ILMerge work when building for netstandard2.0.
Also unify/deduplicate the ILMerge logic between Xamarin.iOS and Xamarin.Mac.
* [msbuild] Build for netstandard2.0 in addition to net461.
* [msbuild] Use custom project configurations to support running the tests for both netstandard2.0 and net461.
Use custom project configurations to support running the tests for when the
tasks assembly is built for netstandard2.0 and net461.
* [tests] Make command-line based 'make test-ios-tasks' run tests for both netstandard2.0 and net461.
* [xharness] Add test configuration to run iOS MSBuild tests using either netstandard2.0 or net461.
* [msbuild] Make the netstandard2.0-buils task assemblies the default.
* [msbuild] ILRepack lib assemblies, not ref assemblies.
Ask MSBuild to copy lib assemblies to the output folder when building for
netstandard2.0, this way we can easily find the actual implementation
libraries to pass to ILRepack.
* [msbuild] Merge System.Text.Encodings.Web.dll as well.
* [xharness] Fix build of MSBuild tests for iOS.
* [xharness] Fix two compiler warnings.
Fixes these warnings:
Jenkins.cs(538,37): warning CS1998: This async method lacks 'await' operators and will run synchronously. Consider using the 'await' operator to await non-blocking API calls, or 'await Task.Run(...)' to do CPU-bound work on a background thread.
Jenkins.cs(877,14): warning CS1998: This async method lacks 'await' operators and will run synchronously. Consider using the 'await' operator to await non-blocking API calls, or 'await Task.Run(...)' to do CPU-bound work on a background thread.
* [xharness] Simplify async code a bit.
* [msbuild] Convert to short-form csproj.
* [msbuild] Make asserts more useful.
* [msbuild] Make tests ignore the actual location of the test assembly.
* [msbuild] Short-style projects default to deterministic builds, which is not compatible with wildcard versions.
* [msbuild] Adjust test.
* Update .gitignore.
* Bump NUnit.ConsoleRunner version.
* [msbuild] Fix indentation.
* [msbuild] Simplify csproj.
The 32bits **debug** binaries are now too big for Apple's native linker
to process, which gives us (non useful) build errors on the bots.
This will still run the release builds configuration of the tests since
they are smaller and still within the limits of the tooling.
Bump mono to get the new splited test dlls and add them to be ran in
xharness. Special logic is used for mscorlib so we make sure that all
the 'parts' of the test dll do have the same configurations.
Co-Authored-By: Waleed Chaudhry <54864665+wachaudh@users.noreply.github.com>
After this commit we will have more application but they will be the
appropiate size so that they can be built with the linker for iOS 32b.
It is important to note that the following apps WILL CONTINUE to fail
since the dlls need to be splitted in mono:
* mscorlib tests
* Mono BCL tests group 5 - Which is monotouch_System.Core_xunit-test.dll
and is too large.
Updating the msbuild tasks to use netstandard2.0 requires us to bump NUnit to 3+.
This requires:
* A few code changes due to breaking API changes in NUnit.
* Changes in xharness and a makefile to cope with the new location for the
NUnit console runner (I added a helper script to make things slightly
easier).
**Problem**
32 bit tests could only be executed with 64 bit tests. E.g: `run-ios-32-tests` didn't work on its own, only `run-ios-tests,run-ios-32-tests` worked
**Solution**
- Change the logic so that 32 bit tests are only looking for `IncludeiOS32` and 64 bit tests only `IncludeiOS64`
- To keep `run-ios-tests` and `skip-ios-tests` working, hack `SetEnabled` so `IncludeiOS` also sets `IncludeiOS32` and `IncludeiOS64`.
Otherwise the way the code is currently organised we'd have to use `skip-ios-32-tests` and `skip-ios-64-tests`.
The latest SDK version and the latest OS version does not necessarily have to
match (for instance the iOS 13.2 SDK can support both iOS 13.2 and iOS 13.3),
so keep track of them separately.
Also use the latest OS version to determine which simulator to run, instead of
the latest SDK version (Xcode 11.3 ships with the iOS 13.2 SDK but only has an
iOS 13.3 simulator, not an iOS 13.2 simulator).
Fixes https://github.com/xamarin/maccore/issues/2066.
* Bump mono to a hash with archives and use them.
New commits in mono/mono:
* mono/mono@6af4ae7635 [2019-06][ci] Add Xcode 11.2beta2 for XI/XM Mono SDK builds
Diff: 476d72b9e3..6af4ae7635
* Bump mono to get min iOS version fix.
New commits in mono/mono:
* mono/mono@3775d5ac0a [sdks] Bump min iOS version to 7.0.
Diff: 6af4ae7635..3775d5ac0a
* [xharness] Bump mtouch tests timeout to 3h, we have a couple of new PR bots which are old and slow.
The new PR bots are late 2012 mac minis, so quite slow.
* Implement a different escaping/quoting algorithm for arguments to System.Diagnostics.Process.
mono changed how quotes should be escaped when passed to
System.Diagnostic.Process, so we need to change accordingly.
The main difference is that single quotes don't have to be escaped anymore.
This solves problems like this:
System.ComponentModel.Win32Exception : ApplicationName='nuget', CommandLine='restore '/Users/vsts/agent/2.158.0/work/1/s/tests/sampletester/bin/Debug/repositories/ios-samples/WorkingWithTables/Part 3 - Customizing a Table\'s appearance/3 - CellCustomTable/CellCustomTable.sln' -Verbosity detailed -SolutionDir '/Users/vsts/agent/2.158.0/work/1/s/tests/sampletester/bin/Debug/repositories/ios-samples/WorkingWithTables/Part 3 - Customizing a Table\'s appearance/3 - CellCustomTable'', CurrentDirectory='/Users/vsts/agent/2.158.0/work/1/s/tests/sampletester/bin/Debug/repositories', Native error= Cannot find the specified file
at System.Diagnostics.Process.StartWithCreateProcess (System.Diagnostics.ProcessStartInfo startInfo) [0x0029f] in /Users/builder/jenkins/workspace/build-package-osx-mono/2019-08/external/bockbuild/builds/mono-x64/mcs/class/System/System.Diagnostics/Process.cs:778
ref: https://github.com/mono/mono/pull/15047
* Rework process arguments to pass arrays/lists around instead of quoted strings.
And then only convert to a string at the very end when we create the Process
instance.
In the future there will be a ProcessStartInfo.ArgumentList property we can
use to give the original array/list of arguments directly to the BCL so that
we can avoid quoting at all. These changes gets us almost all the way there
already (except that the ArgumentList property isn't available quite yet).
We also have to bump to target framework version v4.7.2 from v4.5 in several
places because of 'Array.Empty<T> ()' which is now used in more places.
* Parse linker flags from LinkWith attributes.
* [sampletester] Bump to v4.7.2 for Array.Empty<T> ().
* Fix typo.
* Rename GetVerbosity -> AddVerbosity.
* Remove unnecessary string interpolation.
* Remove unused variable.
* [mtouch] Simplify code a bit.
* Use implicitly typed arrays.
This has a couple of advantages:
* It makes it easier to add a catalyst version of these libraries (because it
becomes cumbersome to build for catalyst when the build rules assumes we're
building for both simulator and device).
* It makes it easier to create an xcframework of our libraries, because the
contents in an xcframework is split like this.
* macOS 10.15 starts putting up permission dialogs we can't automatically
dismiss anymore, so start honoring the 'IncludeSystemPermissionTests' option
for macOS tests.
* Improve the 'IncludeSystemPermissionTests' option to have three states: if
set (either true or false), that takes precedence, but if not set, we now
don't run any tests that require permission dialogs on macOS or on device if
we're running in CI. Tests executed locally will still put up dialogs, both
on macOS and on device.
* This needed a few changes to the html report, since the
'IncludeSystemPermissionTests' is exposed in the UI and the code didn't
handle the three different states.
* Update a few tests to check for permission to the contacts.
Fixes https://github.com/xamarin/maccore/issues/1856.
We have tests whose behavior changes when executed on CI, and those tests use
the BUILD_REVISION variable to detect where they're being executed. For this
to work we need to propagate the BUILD_REVISION variable to the test
executable.
Hopefully fixes https://github.com/xamarin/maccore/issues/649 now.
Trying to find a simulator will mark the test as a failure if the simulator
couldn't be found, and we don't want that to happen to ignored tests.
This should fix an issue where xharness seems to try to run the 32-bit
simulator tests when asked to run only device tests.
We have tests whose behavior changes when executed on CI, and those tests use
the BUILD_REVISION variable to detect where they're being executed. For this
to work we need to propagate the BUILD_REVISION variable to the test
executable.
Hopefully fixes https://github.com/xamarin/maccore/issues/649 now.
This boils down to the makefile-generation code having the information it
needs (and that information being correct).
This fixes running of tests on other macOS bots (older/newer), because now we
can build the test package again.
Manuel de la Pena
Sebastien Pouliot
Rolf Bjarne Kvinge
Pramit Mallick
Alex Soto
monojenkins
Vincent Dondain