Ignore a failing test on devices. Related mono issue: https://github.com/mono/mono/issues/19308
Mono has to fix the issue, so we ignore in the meantime.
Co-authored-by: Rolf Bjarne Kvinge <rolf@xamarin.com>
Refactor result parsing and test it. Make sure that all the possible
corner cases are taken into account. The changes are as follow:
1. Move all the logic to a new class, which will take care of the reustl
parsing.
2. Simplify the parsing method by splitting the noise code to other
functions. There is no need to parse several files in the same method.
3. Add tests that make sure that the correct result is returned when:
a. The app cannot launch.
b. The app timesout.
c. The app crashes.
The _nice_ thing about `NSURLProtocol` is that it does not force the use
of the network - but we inflicted this upon us on tests.
So this updates the test to avoid the network, which should make it
more reliable and not so much influenced by the URL used or the OS
versions where it's executed.
This does not cover as much callback as before. However this it does
test that the NSUrlProtocol is registered correctly (what we care) and
the rest is largely only related the the OS (which is not the target
of our unit tests).
Hopefully fix https://github.com/xamarin/maccore/issues/1956
This is not a complete solution, because it doesn't work when running device
tests, because then we don't build locally. On the other hand, that bug has
never been a problem for device bots, so hopefully that won't change in the
future.
Ref: https://github.com/xamarin/maccore/issues/2177
* [mtouch/mmp] Only use stdout from xcrun as the path of the file that was found. Fixes#8147.
xcrun can print stuff to stderr even if it succeeds and prints what it found
to stdout, which means we have to separate stdout and stderr when capturing
xcrun's output.
This also means that we need a few changes to our RunCommand API:
* Add support for capturing stdout and stderr separately by providing
different callbacks/StringBuilder instances.
* Use overloads instead of default arguments, because with default arguments
we end up with ambiguous matches in a lot of places with the new overloads.
Fixes https://github.com/xamarin/xamarin-macios/issues/8147.
* [tests] Fix mmptest build.
* [mtouch/mmp] Fix process log output.
Bump mono to 2020-02 head to be in sync with Android.
Co-authored-by: Imran Hameed <imhameed@microsoft.com>
Co-authored-by: Sebastien Pouliot <sebastien.pouliot@microsoft.com>
Co-authored-by: Ankit Jain <radical@gmail.com>
Co-authored-by: Manuel de la Pena <mandel@microsoft.com>
Co-authored-by: Sebastien Pouliot <sebastien.pouliot@gmail.com>
First step to try and clean up the psring of results and ensure that we
are doing the correct thing. We have several issue there. We want to
move all the complicated logic out of AppRunner and test the different
outputs and the full result matrix.
This commit first allows to use an instance class for the parsing, later
we will move things out. Step by step
XHarness TestAssemblyDefinition is very particular on the way we deal
with the mono test assemblies, in the general case, it only contains the
assembly name, but in the xamarin-macios it calculates the name of the
assembly using the platform.
Create an interface so that the command line was its own implementaiton
and move as much as possible to use the AssemblyLocator class. Later we
can add two simpler implementations for the more general cases.
Move the AssemblyLocator out of the Templates namespace since it makes
more sense.
* Unify target framework code between mtouch and mmp.
* Simplify the code in mmp: have three possible valid target frameworks for
most of code, and add special code to handle setting any other valid target
frameworks to redirect to one of those three valid target frameworks (and
warn if given any of those valid, but not "main", target frameworks). Any
other code can then depend on the target framework having exactly one of
those specific values, which means we can make IsUnified* variables
convenience properties instead.
* Unify a bit more of the argument parsing code between mtouch and mmp, since
that made a few other things easier.
* Add TargetFramework.IsValidFramework to have one validation implementation.
* Move the implementation of TargetFramework.MonoFrameworkDirectory to mmp
itself, it's not really related to the target framework.
* Remove Driver.IsUnified and IsClassic from mmp, they're not used anymore.
* Formally deprecate --xamarin-[full|system]-framework in mmp, they've really been deprecated for many years.
* Remove LinkerOptions.TargetFramework, it's not used anymore.
* Get rid of mmp's userTargetFramework fried, it's duplicated with the
targetFramework field.
* Add a few tests, and tweak others a bit.
Breaking changes:
* Both mtouch and mmp require --target-framework now. The only direct
consumers should be the MSBuild tasks, which already pass --target-framework
all the time. This simplifies code, and removes assumptions.
Teach the linker about more opcodes it can't compute a constant value for.
This way a debug version of mmp/mtouch doesn't print out these opcodes for
someone to check.
This code is only executed when debugging mtouch/mmp in the IDE, so no change
to released bits.
Initially the location of the test dlls of the downloaded artefact and the compiled mono were different. That was no longer the case and therefore there was a lot of code that could be removed since the wasDownloaded variable was set to be always true.
We remove the not needed code that will calculate a diff path if mono was compiled and use the AssemblyLocator class to point to the correct root path were test dlls will be found.
Co-authored-by: Přemek Vysoký <premek.vysoky@microsoft.com>
At this point the BCLTestProjectGenerator class just represents a configuration class that knows about the project that are particular to xamarin-macios. All the logic that creates the projects has been moved to the XamariniOSTemplate which creates the project using the information passed by the config class.
At this stage, we have the ability to generate test projects by simply passing the info classes, which can be built either by a class (like the generator one) or a command line. We have fully decoupled the project generation from Xharness.
Tests have been added or moved to the correct location.
* Make everything in src/ build into a directory dependent upon BUILD_DIR.
That way the API comparison can successfully build into a different
directory.
* Use an absolute path as the temporary build directory. This works better
when the path is used in directories with different depth from the root.
Fixes https://github.com/xamarin/maccore/issues/2176.
The name of the tests is picked up in a more inner node than
expected. Fwd the test name to make sure that noe all
crashes/timeouts/launches have the same name.
Add extra information when we have a timeout. We want to know the
name and the bot used to ensure that it is easier to identify patterns.
Timeout tests now have:
test title as: App Timeout {AppName} {Variation} on bot {device_name}
message: AppName} {Variation} Test run timed out after {timeout.TotalMinutes} minute(s) on bot {device_name}.
Build issues now have:
test title: App Build {projectTask.TestName} {projectTask.Variation}
That way we know the exact app that failed to build and the exact app that timeout and the device used to run it.
* Don't validate the TargetFrameworkIdentifier, instead pass it along to bgen
(which will validate it). This means less validation updates when something
changes.
* For Xamarin.Mac's GenerateCommandLineCommands' implementation:
* We'll always have a bgen.exe, so the code for what to do if bgen isn't
there can be removed.
* There's no need to compute the 'isMobile' value, because now it's not used
anymore. So we can remove the first if block in the method completely.
* There's no need to add the -stdlib flag, because the base implementation
in BTouchTaskBase already adds it.
* Now there's nothing left in the method but calling base, so the entire
override can be removed.
* This also removes the need for the FrameworkRoot parameter to the task,
so remove that too.
* For Xamarin.Mac's GetTargetFrameworkArgument's implementation:
* Move the TargetFramework logic to detect the different Xamarin.Mac
variations to the targets file, so that it can be reused by other tasks
and targets.
* This means we don't need an overridable function to get the target
framework argument, so just remove inline the entire virtual method.
Also print out something to the system log. In theory
xamarin_process_managed_exception should also end up terminating the process
in one way or another, but when something goes wrong it tend to go very, very
wrong, so this makes sure some info is printed somewhere.
Přemek Vysoký
Manuel de la Pena
Rolf Bjarne Kvinge
Sebastien Pouliot
Waleed Chaudhry
Jo Shields
Marius Ungureanu