Moved the implementations under Hardware. Add the following to simplify
the testing of the AppRunner:
1. Add interfaces for the listing/loading of devices and simulators.
2. Add a special class for mlaunch args. That way we are sure that
arguments are correctly passed and we do not have typos.
3. Added tests for the new classes and for the device and devices
classes.
4. Added tests for the management of the TCC db.
Tests for the SimDevice and Simulators will come in a following PR. The
only reason for it is that I realize that the commit was getting to
large.
Co-Authored-By: Rolf Bjarne Kvinge <rolf@xamarin.com>
Co-Authored-By: Přemek Vysoký <premek.vysoky@microsoft.com>
Move all the extension methods to a class. After this refactor, we will
be able to DI the manager in the other classes and assert that the
processes are called with the correct parameters without the need of
launching them.
Also added tests for the manager. We create a dummy console app that
will be executed by the tests. The console app has a number of
parameters that will be used to ensure that the new process behaves as
we want:
- Use the passed exit code.
- Create child proecesses if needed.
- Sleep to force a timeout.
- Writer messages to stdout and stderr.
Our tests call the dummy app and ensures that the results match the
behaviour expected by the dummy app.
Co-authored-by: Manuel de la Pena <mandel@microsoft.com>
Co-authored-by: Premek Vysoky <prvysoky@microsoft.com>
Co-authored-by: Rolf Bjarne Kvinge <rolf@xamarin.com>
* [Harness] Refactor process management to be testable.
Move all the extension methods to a class. After this refactor, we will
be able to DI the manager in the other classes and assert that the
processes are called with the correct parameters without the need of
launching them.
Also added tests for the manager. We create a dummy console app that
will be executed by the tests. The console app has a number of
parameters that will be used to ensure that the new process behaves as
we want:
- Use the passed exit code.
- Create child proecesses if needed.
- Sleep to force a timeout.
- Writer messages to stdout and stderr.
Our tests call the dummy app and ensures that the results match the
behaviour expected by the dummy app.
* Apply suggestions from code review
Co-Authored-By: Přemek Vysoký <premek.vysoky@microsoft.com>
* Move out utils into a separate namespace
* Move Cache to the test project
* Fix namespaces after merge
* Remove unneeded code
* Move Target files
* Sort csproj
* Refactor Targets
* Rename Cache to TempDirectory
* Fix using
* Move ProjectFileExtensions
* Remove dead code
* Move Extensions
* Add empty StringUtilsTests
* Add StringUtils tests
* Revert refactorings
* Update tests/xharness/Utilities/StringUtils.cs
Co-Authored-By: Rolf Bjarne Kvinge <rolf@xamarin.com>
* Update tests/xharness/Utilities/StringUtils.cs
Co-Authored-By: Rolf Bjarne Kvinge <rolf@xamarin.com>
* Add better formatarguments test
* Update tests/xharness/Utilities/StringUtils.cs
Co-Authored-By: Rolf Bjarne Kvinge <rolf@xamarin.com>
Co-authored-by: Manuel de la Pena <mandel@microsoft.com>
Co-authored-by: Premek Vysoky <prvysoky@microsoft.com>
Co-authored-by: Rolf Bjarne Kvinge <rolf@xamarin.com>
This also meant adding support for running .NET tests in xharness. Some
refactoring was done to extract common code to shared members, in order to
avoid duplicating a lot of code.
Move all the extension methods to a class. After this refactor, we will
be able to DI the manager in the other classes and assert that the
processes are called with the correct parameters without the need of
launching them.
Also added tests for the manager. We create a dummy console app that
will be executed by the tests. The console app has a number of
parameters that will be used to ensure that the new process behaves as
we want:
- Use the passed exit code.
- Create child proecesses if needed.
- Sleep to force a timeout.
- Writer messages to stdout and stderr.
Our tests call the dummy app and ensures that the results match the
behaviour expected by the dummy app.
Co-authored-by: Přemek Vysoký <premek.vysoky@microsoft.com>
Provide tests that will ensure that the Tcp listener and File listener
will work correctly. Tests ensure that both listeners receive the data
and write it in the ILog
The tcp test is an interesting one because it redirects the call of the
method to a diff callback that will write the data in a stream to check
the final result.
Move to use interfaces, that will let us later add tests that will
verify that all the correct logging is performed. As an example, added a
test for XmlResultParser that ensures that the failures are correctly
generated. The test uses Moq to pass the different paths to be used and
later be able to verify the wirtten xml.
Co-authored-by: Přemek Vysoký <premek.vysoky@microsoft.com>
Move to use interfaces, that will let us later add tests that will
verify that all the correct logging is performed. As an example, added a
test for XmlResultParser that ensures that the failures are correctly
generated. The test uses Moq to pass the different paths to be used and
later be able to verify the wirtten xml.
Some crashes are reported as tcp connection issues because they happen
before the app had the chance to write anything in the log. In that
case, we checked for the presence of the file, and if not present we
decided it was a tcp issue when it is not the case.
In this commit, we check for the tcp erorr message in the main_log so
that we are certain that the issue was with the connection and not
anyother.
Testingis simple, ran tests without the phone being part of the same
network and test with a branch that has a crash. For example the one in
https://github.com/xamarin/xamarin-macios/pull/8009 hash 83240612e8
We support different outputs, lets add the avility for the caller to
decide which one to use. We default to NUnit V3 due to or dependency to
it in VSTS.
We do not longer compile the bcl tests, but we consume them as part or
mono. The old BCL targets and classes are not needed. As you can see,
the lists were empty.
Add tests to make sure that we can be safe when we make changes to that
part of the code.
Some methods are not tested but due to dependencies. Will work un
separating things a little to keep moving fwd.
We initially added a cmd to do the bcl test import generation that has
never been used.
We are moving the bcl test importer code to xharness and creating a
NUnit test project to run the tests that we already added. Unit tests
pass.
It will be nice to be able to have a smaller set of tests to be ran by
xharness. In this commit we add a set of labels to be able to divide the
test in the following groups:
* Xamarin - all tests BUT monotouch.
* Monotouch - just monotouch tests.
* Old BCL - The BCL tests that are based on NUnit
* New BCL - The BCL tests that are based on xUnit.
* mscorlib - Just the test that exercise mscorlib (and the different
groups, mscorlib 1, mscorlib 2 etc..)
This will allow to parallelize the execution of the full test suit in
different agents in VSTS.
Xml should be:
```
<failure>
<message>Foo</message>
<stack-trace>Bar</stack-trace>
</failure>
```
But we generate:
```
<failure>
<message>Foo
<stack-trace>Bar</stack-trace>
</message>
</failure>
```
Makes the parsing of the failures impossible.
In order to simplify the monitoring job add the device name to the
following failures:
* Installation
* Launch
* Tcp Connection
All the above are most of the time due to a misconfigured device. The
device name is useful information for the monitoring person to be able
to reach IT and address the issue.
The parsing code set the crashed variable, but it was not set as an
output variable, that meant that the value was not used. Further in the
code we use the variable to decide if we had a crash or not. Most of the
time, there is no probel since if we have a real crash, we will get the
crash reports and to the right thing, but in the case of tcp connection
issues we do not have them, and therefore the crash xml result is not
created.
Bonus: the local logs variable was hidding a variable in another scope,
fixed that.
xharness does not only allow to run tests in CI, but also helps to run
specific project. That feature is used for the mtouch tests. In that
scenario, we do not have a build task and therefore we will not have the
build logs.
fixes: https://github.com/xamarin/maccore/issues/2154
If we are in VSTS and we want to have correct reportings we need to
generate a test failure when the applicaiton times out. If not, we have
a missmatch between the results from xharness (we have failures) and
VSTS (success).
On devices that cannot reach the host via TCP we do not have a log, this
means that in the if statement needs to have a case for it.
The main problem is that when the device cannot connect to the host, we
do not get a log OR a crash reason from the crash logs. It makes sense
not to have a crash reason, because the app did not crash. In these
sitations, we have to create a xml crash report (since we really do not
know if we can parse the file) that will tell vsts that there was an
issue. Adding the main log will let the monitoring person see the
results of the test run.
Because the PublishTest task is taking a regular expression, it is
importing the test results more than one, which gives wrong stats. Add a
prefix to better filter those files we are interested in (thos with
attachments).
As a bonus, refactored the xml failure code for less copy pasting and to
have a single place where we had to add the prefix.
VSTS does not provide a good way to report an app installation issue,
but we can fake a failure in the test when the installation does not
happen.
In the case of the installation failure a xml test result is generated
that will expose all the required information and will attach all the
needed logs (install logs). An example of the generated result can be
seen here: https://gist.github.com/mandel-macaque/2274bcd8785eebd636b98142e228afa9
As per the documentation of the VSTS test uploader (https://docs.microsoft.com/en-us/azure/devops/pipelines/tasks/test/publish-test-results?view=azure-devops&tabs=yaml#attachments-support)
the tests support the addition of attachments. We want to be able to
access the logs of the different tests runs, this is achieved the
following way:
1. Move TouchUnit to use NUnit V3 which allows to add attachments.
2. Once the tests are ran, add the attachments to the following nodes:
1. The very first test-suite. This will allow to have the logs for
succesul tests.
2. Add logs to failing tests. Reduces the number of clicks to be done
to access to the logs when a test case fails.
3. Modify the assembly name of the test-suit to match the name of the
application. This ensures two things.
1. We have a consistent name for the file column in VSTS, that can be
used to see recurrent failing tests.
2. The name is more readable, since if not, it will contain the UUID
of the device.
Logs are not added to succesful tests because it will have the following
problems:
* Larger data storage usage.
* Longer upload time. The addtion of the logs per tests (succesful or
failed) was tested and resulted in an upload time LONGER than 6 hours
for all TouchUnit, NUnit blc tests and xUnit bcl tests.
In order for this to be useful, the task in the pipeline SHOULD NOT
merge test runs. We should have a test run PER application so that we do
not mix the logs.
Co-Authored-By: Rolf Bjarne Kvinge <rolf@xamarin.com>
One of the reason why we lost the TestReport was because the xml added
from the new code did not use the same name for the log as the expected
one (https://github.com/xamarin/xamarin-macios/pull/7835/files#diff-03773861bcef485bf343300f31b60b0eR374).
If we are going to be using the description as a way to decide what is
going to be done with it, lets use static vars so that we do not have
logs added with the wrong name.
Add code to write the TestReport when we have xunit results. Also
refactored code for the NUnit format to not use LINQ, it will use less
memory and in the future we can move to async (not now since it might
raise other problems).
fixes: https://github.com/xamarin/xamarin-macios/issues/7826
Co-authored-by: Manuel de la Pena <mandel@microsoft.com>
As per the documentation of the VSTS test uploader (https://docs.microsoft.com/en-us/azure/devops/pipelines/tasks/test/publish-test-results?view=azure-devops&tabs=yaml#attachments-support)
the tests support the addition of attachments. We want to be able to
access the logs of the different tests runs, this is achieved the
following way:
1. Move TouchUnit to use NUnit V3 which allows to add attachments.
2. Once the tests are ran, add the attachments to the following nodes:
1. The very first test-suite. This will allow to have the logs for
succesul tests.
2. Add logs to failing tests. Reduces the number of clicks to be done
to access to the logs when a test case fails.
3. Modify the assembly name of the test-suit to match the name of the
application. This ensures two things.
1. We have a consistent name for the file column in VSTS, that can be
used to see recurrent failing tests.
2. The name is more readable, since if not, it will contain the UUID
of the device.
Logs are not added to succesful tests because it will have the following
problems:
* Larger data storage usage.
* Longer upload time. The addtion of the logs per tests (succesful or
failed) was tested and resulted in an upload time LONGER than 6 hours
for all TouchUnit, NUnit blc tests and xUnit bcl tests.
In order for this to be useful, the task in the pipeline SHOULD NOT
merge test runs. We should have a test run PER application so that we do
not mix the logs.
Co-Authored-By: Rolf Bjarne Kvinge <rolf@xamarin.com>
One of the reason why we lost the TestReport was because the xml added
from the new code did not use the same name for the log as the expected
one (https://github.com/xamarin/xamarin-macios/pull/7835/files#diff-03773861bcef485bf343300f31b60b0eR374).
If we are going to be using the description as a way to decide what is
going to be done with it, lets use static vars so that we do not have
logs added with the wrong name.
Add code to write the TestReport when we have xunit results. Also
refactored code for the NUnit format to not use LINQ, it will use less
memory and in the future we can move to async (not now since it might
raise other problems).
fixes: https://github.com/xamarin/xamarin-macios/issues/7826
Only one to start... it's been discussed before but we generally
found other ways to do them. Let's continue to pick the best place
but we now have more options :)
We ping the tcp listener to know that we have a tcp connection, that is
written in the xml logs, which means that parsing will not work. Ignore
the ping, parse xml, and make sure that the xml that will be consume by
vsts is valid.
This PR also fixes https://github.com/xamarin/maccore/issues/827 which does not longer happen.
Looks like we have issues with the internet sharing in the VSTS bots,
this means that now that we always try to parse the XML on CI, we get an
exception, catch it and do not show the results.
The workaround simply tries to read the xml, if possible, we will parse
it, else deal with the text only log. The fix will show the results, but
it is a workaround for a configuration issue in the CI.
Unify the harness properties to just look at InCI and remove all the
other ones. There is no real need to have differences between jenkins
and VSTS and Wrench is gone.
We used to test only on Jenkins, and if the build version had jenkins on
it, rather than doing so, just check if we are in the CI by looking if
BUILD_REVISION is present in the env.
Use the xml parsing helper methods to decide if the xml can be parsed
and if we will be able to generate the report. That way we avoid an
exception that makes the CI noise.
Co-authored-by: Manuel de la Pena <mandel@microsoft.com>
Use the xml parsing helper methods to decide if the xml can be parsed
and if we will be able to generate the report. That way we avoid an
exception that makes the CI noise.
Use the xml parsing helper methods to decide if the xml can be parsed
and if we will be able to generate the report. That way we avoid an
exception that makes the CI noise.
We ping the tcp listener to know that we have a tcp connection, that is
written in the xml logs, which means that parsing will not work. Ignore
the ping, parse xml, and make sure that the xml that will be consume by
vsts is valid.
This PR also fixes https://github.com/xamarin/maccore/issues/827 which does not longer happen.
Looks like we have issues with the internet sharing in the VSTS bots,
this means that now that we always try to parse the XML on CI, we get an
exception, catch it and do not show the results.
The workaround simply tries to read the xml, if possible, we will parse
it, else deal with the text only log. The fix will show the results, but
it is a workaround for a configuration issue in the CI.
Unify the harness properties to just look at InCI and remove all the
other ones. There is no real need to have differences between jenkins
and VSTS and Wrench is gone.
We used to test only on Jenkins, and if the build version had jenkins on
it, rather than doing so, just check if we are in the CI by looking if
BUILD_REVISION is present in the env.
We ping the tcp listener to know that we have a tcp connection, that is
written in the xml logs, which means that parsing will not work. Ignore
the ping, parse xml, and make sure that the xml that will be consume by
vsts is valid.
This PR also fixes https://github.com/xamarin/maccore/issues/827 which does not longer happen.
* Acquire permission for the microphone (the microphone permission dialog is
not a blocking dialog, which is why things have worked so far).
* Set the last_modified field in the TCC database to now instead of 1970.
Apparently some permissions time out (kTCCServiceMediaLibrary), which means
that the last_modified field is important to get right.
Looks like we have issues with the internet sharing in the VSTS bots,
this means that now that we always try to parse the XML on CI, we get an
exception, catch it and do not show the results.
The workaround simply tries to read the xml, if possible, we will parse
it, else deal with the text only log. The fix will show the results, but
it is a workaround for a configuration issue in the CI.
Unify the harness properties to just look at InCI and remove all the
other ones. There is no real need to have differences between jenkins
and VSTS and Wrench is gone.
We used to test only on Jenkins, and if the build version had jenkins on
it, rather than doing so, just check if we are in the CI by looking if
BUILD_REVISION is present in the env.
* Bump Xamarin.MacDev.
New commits in xamarin/Xamarin.MacDev:
* xamarin/Xamarin.MacDev@210c664 Adds net451 to Xamarin.MacDev.csproj
* xamarin/Xamarin.MacDev@64db365 [winios] Changes provisioning profiles default path
* xamarin/Xamarin.MacDev@d34430a Switch to short-form projects and build for both net461 and netstandard2.0. (#68)
Diff: 0f578f51e6..210c664e56
* [msbuild] Update to latest Mono.Cecil.
The older version doesn't support netstandard2.0.
No code changes were required.
* [msbuild] Remove unused usings.
* [msbuild] Make ILMerge work when building for netstandard2.0.
Also unify/deduplicate the ILMerge logic between Xamarin.iOS and Xamarin.Mac.
* [msbuild] Build for netstandard2.0 in addition to net461.
* [msbuild] Use custom project configurations to support running the tests for both netstandard2.0 and net461.
Use custom project configurations to support running the tests for when the
tasks assembly is built for netstandard2.0 and net461.
* [tests] Make command-line based 'make test-ios-tasks' run tests for both netstandard2.0 and net461.
* [xharness] Add test configuration to run iOS MSBuild tests using either netstandard2.0 or net461.
* [msbuild] Make the netstandard2.0-buils task assemblies the default.
* [msbuild] ILRepack lib assemblies, not ref assemblies.
Ask MSBuild to copy lib assemblies to the output folder when building for
netstandard2.0, this way we can easily find the actual implementation
libraries to pass to ILRepack.
* [msbuild] Merge System.Text.Encodings.Web.dll as well.
* [xharness] Fix build of MSBuild tests for iOS.
* [xharness] Fix two compiler warnings.
Fixes these warnings:
Jenkins.cs(538,37): warning CS1998: This async method lacks 'await' operators and will run synchronously. Consider using the 'await' operator to await non-blocking API calls, or 'await Task.Run(...)' to do CPU-bound work on a background thread.
Jenkins.cs(877,14): warning CS1998: This async method lacks 'await' operators and will run synchronously. Consider using the 'await' operator to await non-blocking API calls, or 'await Task.Run(...)' to do CPU-bound work on a background thread.
* [xharness] Simplify async code a bit.
* [msbuild] Convert to short-form csproj.
* [msbuild] Make asserts more useful.
* [msbuild] Make tests ignore the actual location of the test assembly.
* [msbuild] Short-style projects default to deterministic builds, which is not compatible with wildcard versions.
* [msbuild] Adjust test.
* Update .gitignore.
* Bump NUnit.ConsoleRunner version.
* [msbuild] Fix indentation.
* [msbuild] Simplify csproj.
The 32bits **debug** binaries are now too big for Apple's native linker
to process, which gives us (non useful) build errors on the bots.
This will still run the release builds configuration of the tests since
they are smaller and still within the limits of the tooling.
Co-authored-by: Sebastien Pouliot <sebastien.pouliot@gmail.com>
The 32bits **debug** binaries are now too big for Apple's native linker
to process, which gives us (non useful) build errors on the bots.
This will still run the release builds configuration of the tests since
they are smaller and still within the limits of the tooling.
Bump mono to get the new splited test dlls and add them to be ran in
xharness. Special logic is used for mscorlib so we make sure that all
the 'parts' of the test dll do have the same configurations.
Co-authored-by: Manuel de la Pena <mandel@microsoft.com>
Co-authored-by: Waleed Chaudhry <54864665+wachaudh@users.noreply.github.com>
Bump mono to get the new splited test dlls and add them to be ran in
xharness. Special logic is used for mscorlib so we make sure that all
the 'parts' of the test dll do have the same configurations.
Co-Authored-By: Waleed Chaudhry <54864665+wachaudh@users.noreply.github.com>
**Problem**
32 bit tests could only be executed with 64 bit tests. E.g: `run-ios-32-tests` didn't work on its own, only `run-ios-tests,run-ios-32-tests` worked
**Solution**
- Change the logic so that 32 bit tests are only looking for `IncludeiOS32` and 64 bit tests only `IncludeiOS64`
- To keep `run-ios-tests` and `skip-ios-tests` working, hack `SetEnabled` so `IncludeiOS` also sets `IncludeiOS32` and `IncludeiOS64`.
Otherwise the way the code is currently organised we'd have to use `skip-ios-32-tests` and `skip-ios-64-tests`.
Co-authored-by: Vincent Dondain <vidondai@microsoft.com>
After this commit we will have more application but they will be the
appropiate size so that they can be built with the linker for iOS 32b.
It is important to note that the following apps WILL CONTINUE to fail
since the dlls need to be splitted in mono:
* mscorlib tests
* Mono BCL tests group 5 - Which is monotouch_System.Core_xunit-test.dll
and is too large.
After this commit we will have more application but they will be the
appropiate size so that they can be built with the linker for iOS 32b.
It is important to note that the following apps WILL CONTINUE to fail
since the dlls need to be splitted in mono:
* mscorlib tests
* Mono BCL tests group 5 - Which is monotouch_System.Core_xunit-test.dll
and is too large.
After this commit we will have more application but they will be the
appropiate size so that they can be built with the linker for iOS 32b.
It is important to note that the following apps WILL CONTINUE to fail
since the dlls need to be splitted in mono:
* mscorlib tests
* Mono BCL tests group 5 - Which is monotouch_System.Core_xunit-test.dll
and is too large.
Updating the msbuild tasks to use netstandard2.0 requires us to bump NUnit to 3+.
This requires:
* A few code changes due to breaking API changes in NUnit.
* Changes in xharness and a makefile to cope with the new location for the
NUnit console runner (I added a helper script to make things slightly
easier).
**Problem**
32 bit tests could only be executed with 64 bit tests. E.g: `run-ios-32-tests` didn't work on its own, only `run-ios-tests,run-ios-32-tests` worked
**Solution**
- Change the logic so that 32 bit tests are only looking for `IncludeiOS32` and 64 bit tests only `IncludeiOS64`
- To keep `run-ios-tests` and `skip-ios-tests` working, hack `SetEnabled` so `IncludeiOS` also sets `IncludeiOS32` and `IncludeiOS64`.
Otherwise the way the code is currently organised we'd have to use `skip-ios-32-tests` and `skip-ios-64-tests`.
The latest SDK version and the latest OS version does not necessarily have to
match (for instance the iOS 13.2 SDK can support both iOS 13.2 and iOS 13.3),
so keep track of them separately.
Also use the latest OS version to determine which simulator to run, instead of
the latest SDK version (Xcode 11.3 ships with the iOS 13.2 SDK but only has an
iOS 13.3 simulator, not an iOS 13.2 simulator).
Fixes https://github.com/xamarin/maccore/issues/2066.
* Bump mono to a hash with archives and use them.
New commits in mono/mono:
* mono/mono@6af4ae7635 [2019-06][ci] Add Xcode 11.2beta2 for XI/XM Mono SDK builds
Diff: 476d72b9e3..6af4ae7635
* Bump mono to get min iOS version fix.
New commits in mono/mono:
* mono/mono@3775d5ac0a [sdks] Bump min iOS version to 7.0.
Diff: 6af4ae7635..3775d5ac0a
* [xharness] Bump mtouch tests timeout to 3h, we have a couple of new PR bots which are old and slow.
The new PR bots are late 2012 mac minis, so quite slow.
* Implement a different escaping/quoting algorithm for arguments to System.Diagnostics.Process.
mono changed how quotes should be escaped when passed to
System.Diagnostic.Process, so we need to change accordingly.
The main difference is that single quotes don't have to be escaped anymore.
This solves problems like this:
System.ComponentModel.Win32Exception : ApplicationName='nuget', CommandLine='restore '/Users/vsts/agent/2.158.0/work/1/s/tests/sampletester/bin/Debug/repositories/ios-samples/WorkingWithTables/Part 3 - Customizing a Table\'s appearance/3 - CellCustomTable/CellCustomTable.sln' -Verbosity detailed -SolutionDir '/Users/vsts/agent/2.158.0/work/1/s/tests/sampletester/bin/Debug/repositories/ios-samples/WorkingWithTables/Part 3 - Customizing a Table\'s appearance/3 - CellCustomTable'', CurrentDirectory='/Users/vsts/agent/2.158.0/work/1/s/tests/sampletester/bin/Debug/repositories', Native error= Cannot find the specified file
at System.Diagnostics.Process.StartWithCreateProcess (System.Diagnostics.ProcessStartInfo startInfo) [0x0029f] in /Users/builder/jenkins/workspace/build-package-osx-mono/2019-08/external/bockbuild/builds/mono-x64/mcs/class/System/System.Diagnostics/Process.cs:778
ref: https://github.com/mono/mono/pull/15047
* Rework process arguments to pass arrays/lists around instead of quoted strings.
And then only convert to a string at the very end when we create the Process
instance.
In the future there will be a ProcessStartInfo.ArgumentList property we can
use to give the original array/list of arguments directly to the BCL so that
we can avoid quoting at all. These changes gets us almost all the way there
already (except that the ArgumentList property isn't available quite yet).
We also have to bump to target framework version v4.7.2 from v4.5 in several
places because of 'Array.Empty<T> ()' which is now used in more places.
* Parse linker flags from LinkWith attributes.
* [sampletester] Bump to v4.7.2 for Array.Empty<T> ().
* Fix typo.
* Rename GetVerbosity -> AddVerbosity.
* Remove unnecessary string interpolation.
* Remove unused variable.
* [mtouch] Simplify code a bit.
* Use implicitly typed arrays.
This has a couple of advantages:
* It makes it easier to add a catalyst version of these libraries (because it
becomes cumbersome to build for catalyst when the build rules assumes we're
building for both simulator and device).
* It makes it easier to create an xcframework of our libraries, because the
contents in an xcframework is split like this.
* macOS 10.15 starts putting up permission dialogs we can't automatically
dismiss anymore, so start honoring the 'IncludeSystemPermissionTests' option
for macOS tests.
* Improve the 'IncludeSystemPermissionTests' option to have three states: if
set (either true or false), that takes precedence, but if not set, we now
don't run any tests that require permission dialogs on macOS or on device if
we're running in CI. Tests executed locally will still put up dialogs, both
on macOS and on device.
* This needed a few changes to the html report, since the
'IncludeSystemPermissionTests' is exposed in the UI and the code didn't
handle the three different states.
* Update a few tests to check for permission to the contacts.
Fixes https://github.com/xamarin/maccore/issues/1856.
We have tests whose behavior changes when executed on CI, and those tests use
the BUILD_REVISION variable to detect where they're being executed. For this
to work we need to propagate the BUILD_REVISION variable to the test
executable.
Hopefully fixes https://github.com/xamarin/maccore/issues/649 now.
Trying to find a simulator will mark the test as a failure if the simulator
couldn't be found, and we don't want that to happen to ignored tests.
This should fix an issue where xharness seems to try to run the 32-bit
simulator tests when asked to run only device tests.
We have tests whose behavior changes when executed on CI, and those tests use
the BUILD_REVISION variable to detect where they're being executed. For this
to work we need to propagate the BUILD_REVISION variable to the test
executable.
Hopefully fixes https://github.com/xamarin/maccore/issues/649 now.
This boils down to the makefile-generation code having the information it
needs (and that information being correct).
This fixes running of tests on other macOS bots (older/newer), because now we
can build the test package again.
Currently we execute most of the same logic both during the configure phase
and when running tests, and the Harness.Mac value is only set in the configure
phase.
While it doesn't matter right now, this makes sure there aren't any future
surprises in this area, since otherwise we could end up with different
behavior between the configure phase and when running tests.
Harness.AutoConfigureMac now loads all the mac test projects both when
configuring and running tests, the only difference is that the test projects
that must be generated are only generated when configuring. This means that
the Harness.MacTestProject list contains the exact same test projects both
when configuring and when running tests.
This made it possible to remove logic to clone (mac) test projects the Jenkins
class (since Harness.MacTestProjects contains all the test projects already).
Consolidate logic to generate (mac) test projects:
* First we generate BCL and mono-native projects from their templates.
* Then we generate Full/System variations of any project that needs it.
This way we can remove logic to generate Full/System variations from the logic
to generate BCL/mono-native projects, which means less duplicated (and less
confusing) code.
To this purpose, significant changes were required:
* MacTestProject.TargetFrameworkFlavor has been modified to contain a bit mask
of the variations to generate.
* MacMonoNativeInfo has been significantly simplified, and some of the
generated code has been moved to the actual template instead.
* Some project generation (in MacTarget) to make things work as expected.
Fixes https://github.com/xamarin/xamarin-macios/issues/6322.
* Share code with GetInfoPListInclude to find the same Info.plist nodes, so
that FixInfoPListNode also finds existing nodes whose names isn't
"Info.plist".
* Add support for specifying the new Info.plist name.
Rename TestPlatform fields to be more in line with the rest of the code, and
drop the Unified prefix, since everything is Unified now.
* Unified -> Modern
* UnifiedXM45 -> Full
* UnifiedSystem -> System
* Bump VSMac to 8.1.0.2742 to fix msbuild issues (#6279)
* Bump VSMac to 8.1.0.2742 to fix msbuild issues
This is required to get the support for the msbuild `ToolsVersion`
change from `15.0` to `Current`.
* [tests][msbuild] Fix Binding resources test with updated msbuild
Test failure with updated msbuild and vsmac 8.1:
```
Xamarin.iOS.Tasks.NativeReferencesNoEmbedding("iPhone").ShouldNotUnnecessarilyRebuildBindingProject(True)
Binding project build did not create package?
Expected: True
But was: False
at Xamarin.iOS.Tasks.NativeReferencesNoEmbedding.ShouldNotUnnecessarilyRebuildBindingProject (System.Boolean framework) [0x000a0] in <74b8f7d8a53e40109916d305bb4d7403>:0
at (wrapper managed-to-native) System.Reflection.RuntimeMethodInfo.InternalInvoke(System.Reflection.RuntimeMethodInfo,object,object[],System.Exception&)
at System.Reflection.RuntimeMethodInfo.Invoke (System.Object obj, System.Reflection.BindingFlags invokeAttr, System.Reflection.Binder binder, System.Object[] parameters, System.Globalization.CultureInfo cul
ture) [0x0006a] in <0519fa732e8845b6a809ce9180f541db>:0
```
The test builds the project multiple times. Before the 3rd build, the project
file's timestamp is updated and expects that the binding package will be
rebuilt. But it is not, because the target `_CreateBindingResourcePackage`
doesn't depend on that project file. So, add that to the target inputs.
* [nuget] Use xibuild to run nuget
Fix errors seen during `nuget restore` for tests:
```
Users/builder/jenkins/workspace/xamarin-macios-pr-builder/tests/xammac_tests/xammac_tests.csproj(213,3): error MSB4024: The imported project file "/Library/Frameworks/Mono.framework/External/xbuild/Xamarin/Mac/Xamarin.Mac.CSharp.targets" could not be loaded. Could not find file "/Library/Frameworks/Mono.framework/External/xbuild/Xamarin/Mac/Xamarin.Mac.CSharp.targets"
```
* [xibuild] Fix incorrect mscorlib.dll being used (#6068)
* [xibuild] Fix incorrect mscorlib.dll being used
The `GuiUnit_NET_4_5` project, when built with `xibuild` uses the wrong `mscorlib.dll`.
From https://github.com/xamarin/xamarin-macios/issues/5760#issuecomment-472457202 :
```
- mscorlib.dll is being used from mono/4.5 and the other system assemblies are from mono/4.5-api
- GuiNet* project is built with xibuild
What is happening here is:
xibuild sets[1] `SetToolsetProperty ("TargetFrameworkRootPath", FrameworksDirectory + Path.DirectorySeparatorChar);`
which points to `/Library/Frameworks/Mono.framework/Versions/Current/lib/mono/xbuild-frameworks`.
This causes $(FrameworkPathOverride) to be set[2] to `/Library/Frameworks/Mono.framework/Versions/Current/lib/mono/xbuild-frameworks/.NETFramework/v4.5`,
but that doesn't have a mscorlib.dll, so it gets reset[3] to /Library/Frameworks/Mono.framework/Versions/5.22.0/lib/mono/4.5/.
If we don't set TargetFrameworkRoothPath, then we get `FrameworkPathOverride = /Library/Frameworks/Mono.framework/Versions/5.22.0/lib/mono/4.5-api`,
causing `_ExplicitReference=/Library/Frameworks/Mono.framework/Versions/5.22.0/lib/mono/4.5-api/mscorlib.dll`(correct one) to be used.
```
Fixes https://github.com/xamarin/xamarin-macios/issues/5760
1. https://github.com/xamarin/xamarin-macios/blob/master/tools/xibuild/Main.cs#L209
2. https://github.com/mono/msbuild/blob/xplat-master/src/Tasks/Microsoft.Common.CurrentVersion.targets#L79
3. https://github.com/mono/msbuild/blob/xplat-master/src/Tasks/Microsoft.Common.CurrentVersion.targets#L84
* Revert "Workaround https://github.com/xamarin/xamarin-macios/issues/5760 in generator csproj"
This reverts commit 9bd927bb7f.
The previous commit for xibuild removes the need for this.
* [xibuild] Handle "incorrectly" cased msbuild property names (#6202)
msbuild property names are case insensitive. While generating the custom
app.config, in `SetToolsetProperty(..)` we try to update the property if
it already exists. But the name lookup was case sensitive, thus causing
the lookup to fail, resulting in two entries for the same property name
differing only in case. Eg. `MSBuildSDKsPath` vs `MSBuildSdksPath`.
Fixed to ignore case.
Fixes https://github.com/mono/mono/issues/14765 .
* Bump VSMac to 8.1.0.2742 to fix msbuild issues
This is required to get the support for the msbuild `ToolsVersion`
change from `15.0` to `Current`.
* [tests][msbuild] Fix Binding resources test with updated msbuild
Test failure with updated msbuild and vsmac 8.1:
```
Xamarin.iOS.Tasks.NativeReferencesNoEmbedding("iPhone").ShouldNotUnnecessarilyRebuildBindingProject(True)
Binding project build did not create package?
Expected: True
But was: False
at Xamarin.iOS.Tasks.NativeReferencesNoEmbedding.ShouldNotUnnecessarilyRebuildBindingProject (System.Boolean framework) [0x000a0] in <74b8f7d8a53e40109916d305bb4d7403>:0
at (wrapper managed-to-native) System.Reflection.RuntimeMethodInfo.InternalInvoke(System.Reflection.RuntimeMethodInfo,object,object[],System.Exception&)
at System.Reflection.RuntimeMethodInfo.Invoke (System.Object obj, System.Reflection.BindingFlags invokeAttr, System.Reflection.Binder binder, System.Object[] parameters, System.Globalization.CultureInfo cul
ture) [0x0006a] in <0519fa732e8845b6a809ce9180f541db>:0
```
The test builds the project multiple times. Before the 3rd build, the project
file's timestamp is updated and expects that the binding package will be
rebuilt. But it is not, because the target `_CreateBindingResourcePackage`
doesn't depend on that project file. So, add that to the target inputs.
* [nuget] Use xibuild to run nuget
Fix errors seen during `nuget restore` for tests:
```
Users/builder/jenkins/workspace/xamarin-macios-pr-builder/tests/xammac_tests/xammac_tests.csproj(213,3): error MSB4024: The imported project file "/Library/Frameworks/Mono.framework/External/xbuild/Xamarin/Mac/Xamarin.Mac.CSharp.targets" could not be loaded. Could not find file "/Library/Frameworks/Mono.framework/External/xbuild/Xamarin/Mac/Xamarin.Mac.CSharp.targets"
```
Usually it's when working on beta Xcodes we make mistakes that only show up
when running on older simulators (and devices):
* Missing/wrong availability attributes.
* Tests for new API that don't check the OS if that API is available.
So automatically enable the tests on older simulators for PR builds when using
a beta Xcode.
* [tests] Remove 32-bit Xamarin.Mac tests (both Classic and Unified).
We're removing support for 32-bit Xamarin.Mac apps (#6300), which means we
don't need to run the tests anymore.
This part of the implementation for #6300, I'm starting with the tests because
some of our XM/Classic tests are failing and making the bots unnecessarily red
(since we'll remove XM/Classic support anyway). Also CI will be faster if we
don't run these tests.
* [mmptest] Fix build.
* [tests] Fix build for mono-native-mac.csproj.
* [xharness] Show links to previous test runs in html report in server mode. (#6031)
* [xharness] Change url for server mode.
* [xharness] Show links to previous test runs in html report in server mode.
This makes it much easier to see what failed in a previous test run.
* [XHarness] Show when we have a watch HE0038 error. (#6060)
To simplify the life of the monitorer, if we get a crash with a HE0038
we will propagate the result to the html and will provide a link to the
issue so that it is easy to report it.
* [xharness] Look for HE0038 anywhere in a log file. (#6175)
Most log files now have timestamps, which means that looking for 'HE00380' at
the start of each line won't work anymore.
Fixes an issue where the HE0038 issue isn't detected properly.
This also makes it so that we only parse the log file once (when the test
finishes), instead of parsing every log named 'Run log' every time we produce
the html report.
The change is a pre step to fix https://github.com/xamarin/xamarin-macios/issues/6162
We are removing the not longer child directory bcl-test/BCLTests since
there is not longer any reason for it to exist. The changes make sure
that the tests work as expected in the new directory.
Next step, move all the test apps to their own directory.
Rolf Bjarne Kvinge
Manuel de la Pena
Přemek Vysoký
Alex Soto
monojenkins
Sebastien Pouliot
Pramit Mallick
Vincent Dondain
Ankit Jain
Sebastien Pouliot