If we are in VSTS and we want to have correct reportings we need to
generate a test failure when the applicaiton times out. If not, we have
a missmatch between the results from xharness (we have failures) and
VSTS (success).
On devices that cannot reach the host via TCP we do not have a log, this
means that in the if statement needs to have a case for it.
The main problem is that when the device cannot connect to the host, we
do not get a log OR a crash reason from the crash logs. It makes sense
not to have a crash reason, because the app did not crash. In these
sitations, we have to create a xml crash report (since we really do not
know if we can parse the file) that will tell vsts that there was an
issue. Adding the main log will let the monitoring person see the
results of the test run.
If the pipeline is set on debug, to be able to see any issues, propagate
that to the make calls so that we can also get the information there,
else debugging issues with the pipelines + make is really hard.
Some of the bots fail to do the checkout (miss configuration). The clean
up step is always executed and assumes the pressence of a script, which
will fail since the script is not there.
The script is small, there is no need to add the rm in an extra file
that needs to be checkout.
This removes an extra warning that is set in the pipeline which is noise
when monitoring.
If the bot could no get the provisioning profiles installed, there is no
reason for certain tasks to run since they are all going to fail. This
adds A LOT of noise in the pipeline for the monitoring person to check
when there is no reason.
Because the PublishTest task is taking a regular expression, it is
importing the test results more than one, which gives wrong stats. Add a
prefix to better filter those files we are interested in (thos with
attachments).
As a bonus, refactored the xml failure code for less copy pasting and to
have a single place where we had to add the prefix.
VSTS does not provide a good way to report an app installation issue,
but we can fake a failure in the test when the installation does not
happen.
In the case of the installation failure a xml test result is generated
that will expose all the required information and will attach all the
needed logs (install logs). An example of the generated result can be
seen here: https://gist.github.com/mandel-macaque/2274bcd8785eebd636b98142e228afa9
The NUnit3 xml requires an id for the tests, but xunit does not provide
one. Add an extension object to the xslt that will return a very lame id
so that the test uploader does not complain.
fixes: https://github.com/xamarin/xamarin-macios/issues/7888
This is a series of commits that tweaks the msbuild tests a little bit.
* [msbuild] Match configuration between MyWatchKit2Extension and MyWatchKit2IntentsExtension.
Fixes this warning:
> MTOUCH : warning MT0113: Native code sharing has been disabled for the extension 'MyWatchKit2IntentsExtension' because the --assembly-build-target options are different between the container app () and the extension (--assembly-build-target:@all=dynamiclibrary). [MyWatchKit2Extension/MyWatchKit2Extension.csproj]
* [msbuild] Add the extra argument for the AppWithExtraArgumentThatOverrides test project to all configurations.
That way the test can be built in all configurations (and still test what it's
supposed to test).
* [msbuild] Clean up project files a bit.
* Removed default 'CodesignKey' values from simulator configurations.
* Removed 'MtouchFastDev' values from simulator configurations.
* Removed 'IOSDebuggerPort' values, not needed for these projects.
* [msbuild] Update test projects to use 64-bit architectures.
Also enable bitcode (+llvm) for release device builds for tvOS and watchOS.
* [msbuild] Simplify test project configuration a bit.
* [msbuild] Unify output path.
Using 'TV' and 'TVSimulator' in the OutputPath works fine, but it makes some
of the tests a bit more complex because they need to special-case this test to
find the path to the resulting .app.
* [msbuild] Fix the MyiOSFrameworkBinding test project to have the same assembly name as the project itself.
This simplifies upcoming testing code a bit.
Move to use a template for the DDFun pipeline. Copy the current pipeline
to a template, set some parameters and recreate the pipeline importing
the template.
Don't remove the entire script, because I believe there's code out there that
checks for the existence of the smcs script to determine whether Xamarin.iOS
is installed or not.
As per the documentation of the VSTS test uploader (https://docs.microsoft.com/en-us/azure/devops/pipelines/tasks/test/publish-test-results?view=azure-devops&tabs=yaml#attachments-support)
the tests support the addition of attachments. We want to be able to
access the logs of the different tests runs, this is achieved the
following way:
1. Move TouchUnit to use NUnit V3 which allows to add attachments.
2. Once the tests are ran, add the attachments to the following nodes:
1. The very first test-suite. This will allow to have the logs for
succesul tests.
2. Add logs to failing tests. Reduces the number of clicks to be done
to access to the logs when a test case fails.
3. Modify the assembly name of the test-suit to match the name of the
application. This ensures two things.
1. We have a consistent name for the file column in VSTS, that can be
used to see recurrent failing tests.
2. The name is more readable, since if not, it will contain the UUID
of the device.
Logs are not added to succesful tests because it will have the following
problems:
* Larger data storage usage.
* Longer upload time. The addtion of the logs per tests (succesful or
failed) was tested and resulted in an upload time LONGER than 6 hours
for all TouchUnit, NUnit blc tests and xUnit bcl tests.
In order for this to be useful, the task in the pipeline SHOULD NOT
merge test runs. We should have a test run PER application so that we do
not mix the logs.
Co-Authored-By: Rolf Bjarne Kvinge <rolf@xamarin.com>
Fixes:
> /Library/Frameworks/Mono.framework/External/xbuild/Xamarin/iOS/Xamarin.iOS.Common.targets(1726,3): warning : The Watch App 'MyWatchApp2' has a CFBundleVersion (1.33) that does not match the main app bundle's CFBundleVersion (1.0) [MyWatch2Container/MyWatch2Container.csproj]
> /Library/Frameworks/Mono.framework/External/xbuild/Xamarin/iOS/Xamarin.iOS.Common.targets(1726,3): warning : The Watch Extension 'MyWatchKit2Extension' has a CFBundleVersion (1.0) that does not match the main app bundle's CFBundleVersion (1.33) [MyWatch2Container/MyWatch2Container.csproj]
Fixes:
> MyWatchKit2IntentsExtension/IntentHandler.cs(36): warning MT4174: Unable to locate the block to delegate conversion method for the method MyWatchKit2IntentsExtension.IntentHandler.ResolveRecipients's parameter #2. [MyWatchKit2Extension/MyWatchKit2Extension.csproj]
The method signature probably changed from a beta release to the corresponding
stable release, and this method was never updated.
One of the reason why we lost the TestReport was because the xml added
from the new code did not use the same name for the log as the expected
one (https://github.com/xamarin/xamarin-macios/pull/7835/files#diff-03773861bcef485bf343300f31b60b0eR374).
If we are going to be using the description as a way to decide what is
going to be done with it, lets use static vars so that we do not have
logs added with the wrong name.
Add code to write the TestReport when we have xunit results. Also
refactored code for the NUnit format to not use LINQ, it will use less
memory and in the future we can move to async (not now since it might
raise other problems).
fixes: https://github.com/xamarin/xamarin-macios/issues/7826
Only one to start... it's been discussed before but we generally
found other ways to do them. Let's continue to pick the best place
but we now have more options :)
Use the xml parsing helper methods to decide if the xml can be parsed
and if we will be able to generate the report. That way we avoid an
exception that makes the CI noise.
* [master] Bump mono to pickup needed nunitlite changes.
Commits are:
* [interp] context can be uninitialized for get_resume_state callback (#18535) mono/mono@455cf7d
* [2019-12] [debugger] Native thread not executing managed code considered as terminated (#18504) mono/mono@a979811
* [jit] Compute has_references correctly for gshared types whose constraint is a generic valuetype. Also emit write barriers correctly for these types. (#18562) mono/mono@f9e5a6d
* [2019-12] Bump msbuild to track mono-2019-10 (#18577) mono/mono@01be275
* [runtime] Disable lldb backtrace display on osx, it hangs on attaching in lldb. (#18591) mono/mono@ef8188a
* configure.ac: remove AC_SEARCH_LIBS for libintl (#18595) mono/mono@e55302c
* Bump bockbuild for Pango patch 1e2d68b
* [corlib] Split corlib xunit tests even more for iOS (#18620) mono/mono@8b72dbb
* [2019-12] [merp] MONO_DEBUG=no-gdb-stacktrace shouldn't disable MERP (#18611) mono/mono@4c93e38
* [aot] Avoid inflating gparams with byreflike types during generic sharing. (#18682) mono/mono@0011444
* Update deprecated query parameter to header (#18705) mono/mono@e65846b
* [NUnitLite] Bump nunitlite submodule. (#18733) mono/mono@36073a0
Diff: 2edccc52a7...36073a0c74
Other commits need for this to land"
* Add a MONO define so that we get the extensions written for mono.
* Update the sources on iphone to remove the TextUI which is not used.
* BONUS: Remove an annoying warning when compiling NUnitLite
* Update the templates too which are use by xharness.
```
warning CS0659: 'ResultState' overrides Object.Equals(object o) but does not override Object.GetHashCode()
```
Rolf Bjarne Kvinge
Waleed Chaudhry
Manuel de la Pena
Alex Soto
Sebastien Pouliot
Marius Ungureanu
Jonathan Peppers